00:00:15.599
so this talk is basically about a project that I announced a long time ago
00:00:21.359
in March and I've been working on it uh not as full-time as I would have liked
00:00:26.480
but I I I'm finally at a point where I feel like I can show off some of what I've been doing so I'm going to be doing
00:00:32.279
a little bit of that today but I think more importantly I'm going to talk about uh the architecture what parts of what I
00:00:37.719
did were hard and what sort of the scope of the project is so first of all we have a logo it's
00:00:45.079
cool awesome uh so we have a logo now uh this is cool I'm pretty excited about it took us some time to get this
00:00:51.559
straightened out and talk is over now
00:00:58.680
okay okay so so we have we have a logo and uh believe it or not little things
00:01:04.159
like having a logo really impact uh how much it feels like a real project so pretty excited about this so I want to
00:01:10.520
start by talking about sort of what how we got to here where we are today and uh
00:01:16.479
how that impacts what I've been working on so first of all in case you don't know what tokaido is um I announced a
00:01:22.680
project called rails. apppp which I renamed to tokaido at some point along the lines And the tagline of tokaido was
00:01:30.000
was let's make rails and Os 10 easy again and interestingly this was very controversial um and the reason why it
00:01:36.320
was controversial was that a lot of people who saw my proposal did not think that what I was
00:01:42.399
tackling was a real issue um so this is one comment that someone said as much as I like as I like Yuda he's tackling a
00:01:48.240
non-issue here here is a more uh
00:01:54.159
colorful uh so installing rails in a Mac is largely trivial it's five shell commands depending on the current level
00:01:59.360
of breakage Ruby Jens that's a different story write a Tut write a tutorial on your blog uh and I think what was kind
00:02:05.960
of interesting about this for me this is basically how I
00:02:11.160
felt what was kind of interesting about it for me uh is I don't actually my full-time job is not really building
00:02:17.000
rails apps I write a lot of Ruby code and I write a lot of rails apps but I write a lot more JavaScript code these
00:02:22.560
days and what that ends up meaning is that I end up building new rails apps on new systems for new clients all the time
00:02:29.560
so I don't I don't bootstrap a rail system and then use it for two years so
00:02:34.640
all the pain is in the is in the past every few months I have a whole new system I need to bootstrap and so it's
00:02:40.319
very clear to me that when Homebrew is doing a weird thing or apple shipped a new version of GCC or uh SQL light. H is
00:02:47.360
not on my system or or or or um this stuff happens to me as a relatively Advanced user um and another sort of
00:02:53.879
interesting anecdote uh there's a guy on tc39 who is not a ruby guy at all tc39
00:02:59.319
is the JavaScript committee but he's a badass JavaScript guy uh he has a uh I
00:03:04.840
think a PhD in computer science uh works on on tc39 and he was trying to use octo
00:03:11.319
press and he offhandedly mentioned to me at one of the committee meetings like I can't get acto press working I it used
00:03:17.360
to work and I'm trying to like edit my blog post and it's like completely not working anymore and I was like oh I'm sure I can get it working like Ping me
00:03:23.920
like a few weeks later he pings me on IRC and he's like oh I'm trying to get octo press working here's the error and
00:03:29.000
I'm like oh I'll take a look at it and like two hours later he was like I just give up entirely I cannot do this so as
00:03:35.360
much as you as a as a programmer who at some point went through the pain and got everything working and now your system works so you think everything's great as
00:03:41.840
much as you feel that way in practice huge numbers of people including I am guessing most people who feel like it's
00:03:47.680
not a real problem go through a fairly significant amount of pain and to make matters worse um I know a lot of people
00:03:54.120
like Homebrew I use Homebrew Homebrew is not is not exactly a repeatable reliable package manager so lots of little things
00:04:01.280
about how Homebrew work and change over time could impact the five quick steps that you just paced into your terminal
00:04:07.560
and in fact do impact that especially when you take new operating system upgrades into consideration so here's
00:04:13.720
sort of the overview I'm not going to talk about any of these items because I'm going to go into them in more detail but the overview of what it ended up
00:04:19.680
meaning to build something like tokaido was basically thinking about Dynamic loading versus static loading uh the
00:04:25.560
fact that Ruby normally builds relative uh builds binaries with hard coded absolute load paths um and how to deal
00:04:33.080
how to make statically linked stuff in the first place and then somewhat importantly and this is something that I think people just forgot about uh when
00:04:40.479
they when they were originally thinking about the problem is how do you set it up so that the steps that you say that work in October 2012 will probably work
00:04:47.639
in October 2013 right how do you make it so that every single time there's a new release of Ruby you don't have to go dig
00:04:52.759
up the script and figure out again how to do the entire thing from scratch I mean that would already be better than
00:04:57.960
what people are doing today which is that every sing Le Ruby user has to do it but still it would probably mean that we wouldn't get timely updates of
00:05:04.639
tokaido as there were new past releases because someone would have to go figure out what's going on with lib AML what you are always that at that again etc
00:05:10.840
etc so um for me the last bullet point turned out to be more of a headache and
00:05:16.639
and I think more of a Triumph at the end than a lot of the other parts so let's start with something
00:05:21.960
simple uh require psych um in between those two lines there's a lot of stuff
00:05:27.919
going on that you don't see um but something simple like this you require psych and you get back a true
00:05:34.000
and now you have a psych Miner so what exactly is going on so uh how many people here know what dynamic linking or
00:05:40.960
dynamic loading is and how many people know how it is different from static linking or static loading okay so like
00:05:47.639
half the room so that's already a good step um so OS 10 first of all OS 10's
00:05:53.400
Dynamic linking and dynamic loading are not the same as the dynamic linking and dynamic loading on on Linux they're
00:06:00.080
similar they are they bear a family resemblance but I'm going to show there's some differences um one
00:06:05.560
interesting thing that os10 provides is there's a lot of uh environment variables you can set that will dump out information about what's going on so
00:06:11.639
that actually turned out to be very useful for both figuring out what was going on in the first place and also
00:06:16.919
figuring out whether what I had done to improve the situation was actually successful whether it was effective so
00:06:22.120
one thing you can do is when you run a ruby when you run any program you can say print out all the dynamic libraries that are being
00:06:28.080
loaded and what you can see here is that lib yaml O2 dib is being loaded
00:06:33.240
dynamically so there is a lib yaml dib that's somewhere in the system and a dib is basically just a bundle of executable
00:06:40.280
code that instead of being hardcoded into your executable inside of Ruby it's actually just sitting there on the side
00:06:45.960
in the system somewhere and the system will go find it um I'm not going to talk at all about how the system actually
00:06:51.280
finds it which again is a little bit different from how Linux works and has its own share of craziness mainly
00:06:56.360
because I just decided I'm not going to bother with this at all I'm going to eliminate all the failure scenarios that come from the craziness around
00:07:02.759
dynamically linking code so uh you can also if you look and do the same thing with OP SSL um you get actually a bigger
00:07:10.319
list so there's libz lib crypto lib SSL you can see that it's actually loading a private framework called trust
00:07:15.400
evaluation agent um it's loading Libs again for reasons unknown um and why why
00:07:23.840
does this happen so the reason why it happens that if you go and you look inside of Psych parer doc you'll see
00:07:29.319
some stuff like yamel parser T yamel parser T yam parser initialize that obviously Aaron so Aaron wrote psych
00:07:35.240
Aaron obviously did not write the write libyaml but he wants to be able to in his C code call out to lib yaml so
00:07:42.840
basically the way that you do this in C is you say you tell the Linker hey I need liby Amo please make sure that you
00:07:49.360
include that you remember to tell the program that's going to run later like please load in liby AML so uh instead of
00:07:56.280
having to essentially what you could imagine copying paste all the C code instead of doing that you basically say
00:08:02.520
hey later on the place where you're going to find this code is in libyaml um and there's usually A.H file involved
00:08:09.360
that actually lets the the compiler know what those those declarations are
00:08:15.319
doing um another tool that os10 provides which is pretty cool is called oool and oool lets you see for a given executable
00:08:23.080
unit what dependencies it has so you don't have to actually run the code and see all the all the spew that spits out
00:08:28.919
what's going on you can actually just say hey os1 please tell me what the dependencies of open SSL that bundle are
00:08:35.200
and here you can see the same list right Li SSL lip crypto lib z um there's no crazy private framework but um in fact
00:08:43.240
that is a dependency okay so what actually happens at runtime so what happens at runtime is you load Ruby and
00:08:49.959
when you load Ruby it dynamically links op SSL dynamically linking open SSL is is pretty fine it doesn't really
00:08:56.320
introduce any problems because open SS sl. bundle is just is part of Ruby so Ruby knows where to where to look it up
00:09:02.200
where to find it and it doesn't really introduce any external dependencies it's just a dependency that happens to be a
00:09:08.880
separate file because you don't want to necessarily load every single uh bundle that Ruby generates every single time
00:09:14.440
you load some stuff um these other things lib SSL lib crypto and lib Z are actually external dependencies so each
00:09:20.959
each one of those three things is something that at when you compiled Ruby in the first place it found on the
00:09:26.279
system somewhere and now when you run it again it wants to find it in the same place so that's an external dependency
00:09:31.880
So based on what I just said which is that it expects to find it in the same place you could already start to understand why Distributing a binary
00:09:38.040
distribution of Ruby might be a problem because maybe it found lib Z on my system in a particular location and then
00:09:44.320
it said okay this is where you're going to find it later but then when I give the entire package to you it happens to
00:09:50.000
be in a different location now if you're a Linux guy you're probably thinking hey that's no problem at all you can just uh
00:09:56.800
the the system will take care of it the system will find lib Z that not how OS 10 Works OS 10 needs you to hardcode
00:10:02.440
exactly where the location is into the into the uh bundle there's ways around it with uh special at executable path
00:10:09.320
and some R path stuff but this is a rabbit hole you don't really want to go down OS 10 basically doesn't give you the tools that are necessary and I think
00:10:17.120
uh even if all that worked relying on the user to have installed and put libyaml somewhere in the load path is an
00:10:23.600
external dependency that easily causes pain so I'm sure some of you have seen at some point using something like rvm
00:10:28.760
like could could not install psych does do not have libyaml and that's because psych was not able to find liby Amo and
00:10:34.680
now there's an external dependency you are required to install it first and every so I only showed a few
00:10:40.839
external dependencies Ruby has many more every single external dependency is an opportunity for something to fail so um
00:10:47.399
even something that you would expect to be relatively stable like op SSL uh Apple deprecated op SSL and will
00:10:52.959
probably remove it at some point in the future so you can't really rely on open SSL to exist if you try to compile it you're going to get a bunch of
00:10:58.519
deprecation warnings Apple does all kinds of crazy Shenanigans with it so even relatively stable things apple does not promise
00:11:05.040
will exist forever so you might upgrade to mountain lion try to run something and now whoops open SSL does not exist
00:11:10.920
anymore libz is no longer included in the distribution in a place the Linker can find it so that has not neither of
00:11:16.639
those have happened but open SSL is deprecated so I would imagine in the next release it will probably be
00:11:21.959
removed so here is an interesting another interesting uh anecdote so here's a stack Overflow post how to
00:11:28.160
solve Ruby insulation is psych and someone said hey have you tried to install liby anal following these instructions so first of all this is a
00:11:34.600
fail for me already the fact that someone tried to follow the easy five instructions and got a fail and had to
00:11:40.880
go to stack Overflow obviously bad so this guy provided a link and he said yes I have no luck and that's the end of the
00:11:46.040
threat right so so some guy there's some poor soul
00:11:51.240
who was trying to do something with Ruby and uh was unable to because there was an external dependency that for some
00:11:57.160
reason that it's impossible for us to Divine exactly what failed failed so
00:12:02.360
what is the solution to these problems the solution to these problems is to not rely on comp uh compiler to compile uh
00:12:10.440
and find the the bundle on the or dial on the file system and to not rely on
00:12:15.920
finding it again when you run it so th both of those are that's a brittle Association but even just finding them
00:12:21.720
in the first place in involves things like libel does not exist on my file system how do I make sure it's there in an appropriate place and basically what
00:12:28.560
that ends up like is that instead of Ruby opening open SSL which then uh loads a shared lib SSL you
00:12:36.160
basically just essentially copy and paste lib SSL lib crypto and libz into the open SSL bundle and once upon a time
00:12:43.839
when memory and file uh file space was limited people really didn't like this they wanted to be able to share uh lib
00:12:50.720
SSL across as many parts of the system that used this um when they tried to do that on Windows that ended up with dlll
00:12:57.760
and on most systems where you distribute like an app package either as a MSI or
00:13:02.800
apppp people tend to move away from shared libraries because there's just too much likelihood that something will have gone wrong between the time that
00:13:09.880
you package up your do app and the time that the computer goes to look for it um it could be something like you ran some
00:13:15.600
bad Homebrew uninstallation and it like removed user local or something right like any Anything could happen crazy
00:13:21.279
crazy stuff happens so um basically the idea is that you get lib SSL you get lib
00:13:26.920
crypto you get lib Z and you just put them in the open SSL cell bundle and now you don't someone like me does it for
00:13:32.160
you and now you have it in a bundle and you never have to worry about it again and your system doesn't have to go looking for it and the way that this
00:13:37.920
works in practice is lib SSL lib crypto and lib Z when you compile them you can compile them in a special Mode called
00:13:44.320
Static and it will give you a what is called an archive file a a and the the goal of an archive file is to be able to
00:13:50.360
take the actual binary and again essentially copy and paste it into another binary so that it's not exactly
00:13:56.120
a copy and paste I would not recommend using cat or something like that um but the basic idea is that it's a
00:14:02.160
relocatable binary that you can just put inside of something else um now the cool
00:14:07.240
thing is that once you get this working uh these things you do ool again these things go away and now you have only Li
00:14:14.040
system and lib oby which apple is extremely extremely stable about right so if they made lib system go away that
00:14:19.800
would be extremely bad it has that has not changed since OS 10.0 right
00:14:24.959
definitely stable and if you go and you run uh diali dial print libraries open SSL
00:14:32.519
again you a bunch of stuff goes away and you get to only bundles that actually came with Ruby only bundles that that
00:14:39.000
are part of the Ruby distribution now the problem that I had here is I was like okay so how do I how
00:14:46.399
do I actually statically compile and there's a lot of documentation and good tutorials for Linux and there are fewer
00:14:53.880
good tutorials for OS 10 and when I went to read the Linker uh documentation there are many many lines of it and it
00:15:00.600
changes across different versions of os 10 so uh I'll just save you some time
00:15:06.399
and this is the part that you care about and the part that you care about is that there's a setting that is only the
00:15:13.120
default since xcode 4 so you actually cannot compile tokaido in xcode 3 and
00:15:19.079
basically I will show you how this works but the idea basically is here is how you tell OS 10's Linker to statically
00:15:24.519
compile now if you're again if you're familiar with Linux you're probably familiar with like Dash be static or something like that no this is not how
00:15:30.639
it works in OS 10 the way it works in OS 10 is that when it's when when the Linker is told to find libyaml it is not
00:15:38.120
told whether it wants to do it statically or dynamically it is just told libyaml is in this program is interested in libyaml linking so for
00:15:44.759
every item that is in the Linker path it goes and it says is there a libyaml a if
00:15:51.319
the answer is yes then it's statically links the answer is no it goes looking for a libyaml DB if finds a libyaml dib
00:16:00.040
dynamically linked otherwise go to the next LD path so what you have probably noticed that there is that there is no
00:16:06.440
way to set up a configure to say I would like to statically link Li libyaml that is a thing that is determined purely
00:16:12.959
dynamically when the Linker happens to go looking in the file system so when you want to build a stat a static
00:16:18.519
distribution you have to make sure your path is properly configured or else right it's easy to this is one of these
00:16:24.120
it must be repeatable if you don't make sure that the process that you're using to build it on an OS 10 system make sure
00:16:29.800
that the LD path that contains Thea files is at the front and does not have dils in it you're going to mysteriously
00:16:36.079
start dynamically linking instead of statically linking so this is not desirable this is not what you
00:16:41.720
want now there's another sad story here which is that there is no standard way
00:16:48.680
that you ask when you configure a Unix program to give me aa file there are many many different solutions so libyaml
00:16:56.680
uses disabled shared open SSL no shared without a dash dash uh sqlite you have
00:17:02.720
to people say enable static and disable shared um so you basically and and
00:17:08.640
there's more every sing basically every single one does a slightly different thing and when I found this out this
00:17:14.319
also made me very sad that is a lion face paling um and what what I learned is
00:17:22.240
that I decided to write tokaido at the right time um because there is this thing called SM and no no no SM is not
00:17:28.400
what you're thinking it's the system management framework but this is the logo
00:17:35.559
so anyway so the idea behind the system management framework is to provide a
00:17:42.799
standard way of talking about things like I would like to compile a static version of libyaml so if you go look at
00:17:49.160
the libyaml configuration in in the SM framework there's basically a whole bunch of packages and I would not use
00:17:54.400
this as your standard package manager it's more like a package manager for package managers
00:17:59.919
um what you'll find is that you you what you have the current version of libyaml where to download it from and then if
00:18:05.760
you say please compile is statically what is the flag and um SM doesn't have the static for everything it mostly has
00:18:12.640
them for things that tokaido needs because I went around and said hey can you add this static thing otherwise it assumes it's Dash that static which is
00:18:19.159
basically never the case um but you could but basically this provides a way for us to talk about this it provides a
00:18:24.760
way for us to say in open SSL it's no shared and here are some configur flags that you're going to need right it
00:18:30.080
provides a way for us to talk about it and basically what that means is that there's an easy way to build local.
00:18:36.120
archives for packages using a static flag a flag that's just static and then there's going to be a thing that
00:18:41.880
actually abstracts that away and one of the really awesome things about this whole about using SM is that tokaido
00:18:47.919
distribution itself can be compiled for Linux for Windows for whatever because I'm just using under the hood I'm using
00:18:53.679
the standard uh some standard tools that work in uh environments with a c compiler so um I don't I don't actually
00:19:00.679
know how well it will work on Windows but it definitely works on Linux we've tested it on Linux and obviously the UI
00:19:06.559
Parts don't work but the static distribution works and um SM also manages the ldp pass so SM makes sure
00:19:12.600
that if you have created an archive for something that that is going to be at the front of the LD path when you try to compile something else so this is this
00:19:19.039
is how we manage to do this repeatly repeatedly is that I am not actually in charge of this there's another thing that's in charge of it and it offers me
00:19:25.440
this guarantee um SM is by the rvm guys so rvm they're building the next version of rvm on top of SM um so it's pretty
00:19:34.200
wellmaintained it's they're smart guys it's based on uh shell scripts which uh
00:19:40.240
definitely not my cup of tea but but uh reliable and another really cool thing
00:19:46.400
about all this is that because I worked closely with the rvm guys to use SM um rvm eventually realized that actually
00:19:53.320
reliably compiling Ruby on OS 10 is as hard as it looks like as I as I said it
00:19:59.240
was now rvm always had a slight leg up which is that they could just oh you need libyaml we'll just go download it
00:20:04.840
and put it in a directory and then we'll link against that and we'll hardcode that path and they could do hacks but um
00:20:10.039
over time what rvm realized was that some nonzero percentage of rvm users would simply fail for reasons unknown um
00:20:16.720
they would try to add more hacks but eventually uh rvm realized you know what Yehuda and us are together building this
00:20:23.520
thing called tokaido why don't we just download the binary that we have built together and use that so at some point
00:20:28.799
in the near future when you say rvm use 193 you're going to get a statically linked reliable build that came out of
00:20:35.080
this process which is I think a nice thing there's a SL there's another thing
00:20:40.320
which I sort of handwaved over uh a bit which is that um if you actually do this oool you'll see that it is uh relying on
00:20:48.240
I snipped this out before but um it's relying on a full absolute path to Ruby obviously this is not going to work for
00:20:54.480
me to give you a copy because you don't have a user's ycats and uh absent everyone one creating a user's ycats
00:21:00.320
seems bad so you can see there's a hard-coded path here which is not great and the
00:21:06.360
cool thing is that Ruby has a feature called enable load relative which is completely undocumented but basically what it does
00:21:12.440
is is when you load Ruby it if you have compiled with the enable load relative
00:21:17.880
flag instead of it having hardcoded the exact path it basically says okay where is the Ruby binary and ah the Ruby
00:21:24.600
binary is here so go dot dot go into Li and that's where you should find this standard library and that basically
00:21:30.159
means that there's a little bit more cost to that than hard coding the paths but it makes the whole binary relocatable which is really great um
00:21:37.279
there have been a wide variety of bugs in enable load relative which is why it would be impossible to ship tokaido for
00:21:43.240
Ruby 187 or even 192 um probably the most important bug that exists with a AAL relative is that os10 does not
00:21:51.080
actually provide a mechanism for getting the location of an executable that was not dynamically loaded so if you run
00:21:57.880
Ruby that's not dynamically loaded that's just like the path looked it up somewhere and os10 does not provide a
00:22:03.200
mechanism for getting that so it would always just look at argv and completely fail and what ruby 193 doeses which is
00:22:10.279
pretty awesome is it basically just looks in the path for where Ruby came from and then that's going to be where it is so if you dynamically load Ruby
00:22:17.520
that's fine but we're not doing that here um but if you and if you statically load Ruby then it has a hack basically it looks in the path for where from once
00:22:23.880
it came and basically what that means is that I can't even make tokaido for 192 because I enable Lo relative doesn't
00:22:30.039
actually work on OS 10 on 192 so 193 is actually the first version of Ruby that
00:22:35.080
for all these various reasons actually works reliably and if you actually look at
00:22:42.400
uh if you look at it you'll see that um the inform instead of so why did Ruby
00:22:47.919
want the Ruby to have the dynamically loaded it wants to not have to smash the dial for Ruby inside of the binary
00:22:53.799
basically the solution is if you say enable a little relative it's like well I guess I have no choice I'll just smash the entire Ruby executable into the
00:23:00.080
binary so the actual Ruby binary gets bigger but the only cost is that now you
00:23:05.520
don't have a ruby diive that you're loading it's instead in the Ruby biner it's it's it's not clear to me why this
00:23:11.559
is not the default it's not clear to me why there's a ruby diive okay and now if you uh run oool
00:23:19.360
you we already talked about the fact that we remove lib SSL Li crypto and lib Z and we also now remove Li Ruby which
00:23:25.240
basically means again no external dependencies I handwaved over the LI Ruby one but before so another thing that is so so
00:23:32.799
sort of along these same lines if you actually go to the Ruby bug tracker there's like fix enable load relative on
00:23:38.360
systems with lip 64 I'm pretty sure tokaido is the first time anybody has tried to use enable load relative for
00:23:43.640
anything so M papis who did a lot of the work on building the statically compiled Ruby using SM has been filing a lot of
00:23:49.320
tickets against enable Lo relatives so hopefully hopefully enable load relative will turn out to have worked reliably on
00:23:55.279
193 hopefully I won't have to say he too oh seems great okay so that's basically
00:24:01.360
statically compiled Ruby so now that I'm done with all that now there is a way to actually build a package which is a ruby
00:24:07.240
that doesn't have any hardcoded paths that doesn't have any weird external dependencies that actually works like a regular Ruby so how do we actually get
00:24:13.480
this onto people's system so how do we deliver it so I'm going to just show you a quick demo of what is already working
00:24:19.960
um so let me let me do that real quick so mirror
00:24:26.520
displays so here's xcode going to boot it up and the first time
00:24:32.120
it ever boots up it asks for your password I'll explain why in a bit so you type in your password
00:24:39.440
and ah it's not brought to the front and here's tokaido so nothing really is going on uh here just going to add a
00:24:46.480
going to add an app which is Ember and then I'm going to open it in
00:24:52.000
terminal and come on and now we have we have it
00:24:58.399
open and terminal so obviously there's a little bit of unpolish in some of how this works um but if you go which Ruby
00:25:03.679
you'll see that it's the Ruby that came with application support so um this is basically I consider this the MVP of
00:25:08.960
tokaido of you can add things and there's a statically compiled Ruby that comes with the app package that
00:25:15.279
um that will work and so why did I do it like this why did I ship it as part of
00:25:20.600
uh an application instead of installing in the system so the problem with installing in the system is that it will probably work immediately after you
00:25:25.760
install in the system but it's basically a fixed point in time and if someone screws up the system after that it's
00:25:31.159
basically game over so the cool thing about how this works is that what open a terminal does is it opens a terminal and
00:25:36.480
then once the terminal is open it basically makes sure the path is set to the right place makes sure the gem home is set to the right place right so it
00:25:42.600
basically guarantees that all the environment variables are set correctly um even and and there's no way to screw
00:25:48.200
it up because we're running after your system that may be screwed up is running so um we'll definitely I I don't think I
00:25:54.720
have a slide for this but we're going to have a way to run it as your system Ruby um initially through rvm and possibly through other ways but I think for even
00:26:01.760
for me personally I I like the I like knowing that when I open a terminal window it's it's definitely it's
00:26:08.080
basically guaranteed to work it's not going to have weird system stuff is not going to mess it up um and another thing worth pointing out here is that there's
00:26:14.080
a reason it opens a terminal window I'm not trying to write a guey for rails or a guey for Ruby I'm just trying to make
00:26:19.919
uh a much more much less error prone way of getting Ruby to people's machines that will work reliably um this is a
00:26:25.960
tool that I personally would like to use okay so let me
00:26:33.600
unmirror so that's the quick demo now if you looked closely um you would see that there's a thing there that says ember.js
00:26:39.559
tokaido so what is that about and what it's about is that as part of this process I have also written what is
00:26:45.320
essentially a pure Ruby version of pal I'm going to talk next about how how that is working so the main difference between
00:26:52.399
pow and my pure Ruby version of pow is that my version works with a proc file so instead of it relying on a rack app
00:26:59.559
to boot up it basically will just run the web section of your Brock file with a port and um it will then connect to it
00:27:06.679
so you don't have to if you want to use like Jackal or some other program or if you happen to use foran already uh
00:27:12.440
basically it'll do it'll do the right thing it'll allow you to boot stuff up with a port instead of requiring config Ru I know I personally can't always
00:27:18.720
figure out how to get like a jackal config Ru or a static mtic config Ru it's probably usually possible but it's
00:27:24.240
not always obvious how to achieve that versus I already know how to run this program so please just do it so here's what a proc file looks like um
00:27:31.399
I think this was invented by Heroku and basically the idea is that you just have different sections so here I only have a
00:27:36.640
web section um and Dollar Port is basically just the way of saying I would like to use this port this is also what Heroku uses so if you're already using a
00:27:42.640
proc file for Heroku um everything will everything will work fine with tokaido
00:27:47.840
so let me show a quick demo of the pure version of pal this is not integrated into to tokaido yet so I will ah I
00:27:57.000
appear to have closed seems bad ah here we go okay
00:28:03.960
so so here I'm just going to bootstrap so ah seems very
00:28:09.799
bad okay mirror displays okay so here's my terminal and I'm basically just bootstrapping it you
00:28:15.559
can see I'm setting some environment variables here for where some sockets should go and I'll talk more about this architecture in a minute so you
00:28:21.480
bootstrap and it basically runs a bunch of stuff um and I'm going to open IRB
00:28:27.240
over here
00:28:33.120
okay so let's require socket and then I'm going to get a new socket so basically the idea behind this is that
00:28:39.640
um that this eni this socket is at a place determined by the environment variable and this is something that the tokaido UI will set up for you so
00:28:48.679
obviously seems bad ah I know what is going on
00:28:57.039
here uh so this is in to
00:29:03.679
code tokaido bootstrap slm okay so now I have a
00:29:10.320
socket and I can write I can write to the socket um here I'm writing please
00:29:15.760
add this program that is at some location with this host and this port so I'll do that and you can see over here
00:29:22.080
that it received the message there's obviously more verbose logging this is still in a uh not fully complete and you
00:29:28.440
can see it booted up web rck which is basically what the proc file told it to do which was just to run rack up um and
00:29:34.919
there's actually a bunch of stuff going on here so let's just curl HTTP con we asked it for temp.
00:29:42.360
toid so now we got hello world which is great and if we want to run we can do
00:29:47.880
the same thing we can uh run a different thing and we'll put it at a different port and again you can see got the
00:29:54.679
message and now if we curl temp 2. tokido it works so I I'll talk about the architecture in a second but basically
00:30:01.000
the idea is it's a it's a system that works similarly to Pal except using a brock file and it's written in Ruby I
00:30:06.480
didn't want to I didn't want to have to ship node with tokaido
00:30:12.360
so okay so that's the demo um so how does
00:30:17.440
it work so basically the way it works is when you say give me an app. tokaido it goes into Etsy resolver toid so os10 has
00:30:25.480
this feature called Etsy resolvers and basically you can say inside of etsy resolvers like hey for this domain which
00:30:30.720
is tokaido please go get the name server at this particular Port so there's a project that I created called toid DNS
00:30:37.000
which basically lives at that port and it will say Okay Local Host the answer is Local Host um and then there's a a
00:30:45.320
way to create a firewall rule um in OS 10 which requires pseudo permissions uh pseudo IPFW blah blah blah and that will
00:30:52.440
basically say okay don't go to Port 80 go to Port 2 3 4 56 and that will go to a another project that I created called
00:30:58.960
mxer which is basically doing the proxying which will then say go to Local Host the whatever report you told it and
00:31:05.519
then that will go to your app so that's basically the architecture it's a little bit crazy but it's actually almost identical to Pal's architecture it's
00:31:11.880
just if you want temp. tokaido you have to actually do all these steps um unfortunately so what is how does this
00:31:18.159
actually get delivered so obviously um people are not going to be opening IRB and typing stuff in so basically the way it works is that the first time you load
00:31:24.720
tokaido it's going to uh install a firewall firewall rules launch Damon and
00:31:31.799
basically this launch Damon the reason you need a launch Damon is that IPFW requires pseudo and you don't want to have to be typing in your password every
00:31:37.320
single time you open tokaido so we create a launch Damon that has pseudo permissions that we can talk to um and the way this works is that there's a
00:31:43.919
socket that it listens on and you can uh I'll show you in a second how that works
00:31:49.159
uh and then another thing that that tokaido does when it boots up is it boots up tokaido bootstrap and creates a
00:31:54.600
manager and listens on the tokaido socket so basically it runs on the only the first boot it launches
00:32:00.120
the launch D otherwise it launches the manager and the manager loads tokaido DNS and mxer which are the two programs
00:32:07.399
I talked about and manager is basically in the same memory space as mxer so the first thing now when you boot the
00:32:13.480
application it just sends a message called enable fire all rules sln to the launch Damon which will actually go and
00:32:19.960
now that it has pseudo control it could do enable it can do the IPFW command so the idea here is that unlike pal I don't
00:32:26.360
want this running all the time I only want everything running when you actually have the app open um so that way when you close the app everything
00:32:32.000
continues to work as expected so that's just a philosophical uh tokaido is supposed to be
00:32:37.639
self-contained it's not supposed to take over your system so the next thing so now after it does that now it just goes off it stays
00:32:44.880
off in the side and listens for things at Local Host 80 calls it a day and uh
00:32:51.360
now the the first thing that's going to happen is it's going to go and it's going to send all the it's going to send a bunch of commands to the manager which
00:32:57.120
are basically add these add these applications at these ports it's like a little line protocol and the manager is
00:33:03.600
going to go and add the application which is in Ruby and then event immediately assuming that there were no errors like you didn't try to buy to the
00:33:10.080
same port multiple times or use the same host otherwise you get an error response you're going to get a thing that says added and the idea here is that tokaido
00:33:16.880
can use this information to like show that it's loading right and then at some later Point you'll get a ready message
00:33:22.240
which allows you to show like a green a a green button
00:33:35.039
also go if the user says I actually don't want this app anymore to be on disable it you you can call remove
00:33:40.480
that's going to call remove app and then at some later point it's going to say okay I removed the app and that's going to send removed and that's how uh the
00:33:47.440
application is aware so one really cool thing about this project is that because the point of the project is to have a reliable version of Ruby I can actually
00:33:54.159
write a lot of Ruby code with a lot with not a lot of stress because I know that I can test on the exact Ruby that I'm
00:33:59.519
shipping and if it works I don't have to worry that the user has some crazy Ruby so I I'm writing a lot of sort of
00:34:04.720
network code that would normally give me heartburn like who knows what kind of crazy stuff is going on but I know exactly what kind of crazy stuff is
00:34:11.159
going on because I already shipped it so it's cool so that's that's sort of the architecture of the pal stuff um I think
00:34:18.359
it's pretty cool I think it will I think be people like pal and I think they will like a more scoped version of it I think
00:34:25.560
this this is cool um and finally I want to show some mockups of what the full UI is going to look like so this is what we
00:34:32.240
showed already this was the original mockup it was replaced with a ruby icon which you saw in the quick demo before
00:34:37.839
and you can see the little green icon there and there's like a drop down stuff like that so this is like the new app
00:34:42.960
screen this is a logger so basically specifically for rails apps there will be a visual logger that you can use for
00:34:50.159
to see what's going on and the idea basically is now that you're using tokaido like you would be using pal you
00:34:56.119
don't really have a terminal t open with your log so you would like to see some logs somewhere and I guess we could just
00:35:01.520
dump your logs or we could spend some time to make something that actually adds some value um and I and a I did a
00:35:07.680
lot of thinking about what makes sense to be in a UI and I think I came to the conclusion that people are not particularly attached to their like
00:35:13.200
tailing log workflow so if I provide something that's better than the tailing log workflow I don't think I'm going to
00:35:18.400
get a lot of pain like for example if I added a run this rake task I think people would be very emo about that
00:35:23.520
because I would be taking over something that really belongs in the command line I don't think tailing the log is is a a
00:35:29.160
thing people are particularly attached to so you can see here there's some stuff there's some cool stuff um I think the top thing is going to change because
00:35:35.800
the exact response time in development mode doesn't really matter um but you can see like you can see which which
00:35:41.079
requests came from the parents so if you hit a page and that ask for a bunch of assets you can see that they're nested
00:35:46.280
together um you can see whether things were errors uh how long things took what the whether something was a 500 or 200
00:35:52.000
or whatever and additionally there will be a bunch of notifications so obviously
00:35:57.079
like your p had an error as a notification but also um will periodically run stuff like bundle outdated for you and give you a yellow
00:36:03.400
notification that says like hey this is outdated maybe you care about it and the X will basically be like I actually don't care about this particular bundle
00:36:09.480
outdated and then if it changes you'll get the message again um things like deprecation warnings um things like
00:36:16.359
maybe even like code climate right so things like running uh code code uh Clan
00:36:21.839
against your app and then telling you like hey you should maybe look at this here's some code to look at basically
00:36:27.000
just having a a place to put notifications that are either urgent the red or more just like this might be
00:36:33.680
something you're interested in and I think we won't initially but we'll eventually have a plugin architecture so people can add their own notifications
00:36:39.880
like the compass deprecation warning we probably won't ship with but should be easy to add um also this screen um is
00:36:46.920
it's the only part of the OS 10 app that will be written as a UI web View and again the idea is that if you're SASS
00:36:51.960
and you want to provide your own logging view that's not going to be the exact same thing as a rails view you should be able to drop that in there'll be some
00:36:57.280
kind of JavaScript to OS 10 protocol for like you'll get notified when there's a
00:37:02.319
request that came in ETC and then finally this is just the uh creating the application editing screen
00:37:08.760
and the idea here is you can do stuff like choose an icon or we'll find one for you or whatever that's great um change the host name uh you can see it
00:37:15.400
tells you what if you want to share so one cool thing here is that because every single app has their own port it's
00:37:20.599
easy to share over a local network like hey check this out and it tells you like austin. local colon blah um you can
00:37:27.160
choose Auto assign a public port or not um we also want to support people who don't want to use tokaido Ruby sandbox
00:37:32.800
so like if you want to say I'll just use rvm that's great we want the UI to work for you um and then we also want to have
00:37:38.960
these settings should be should be configurable via yaml file so that people can say like I always want this
00:37:44.160
icon I always want uh Auto sign public Port I I want this to be my application name and I have that checked into uh
00:37:50.280
Source control so everybody's basically in the on the same page so that's basically that's basically where we're
00:37:55.680
heading I think we'll probably have something in between what we have now and this in within a month or so so I
00:38:01.839
don't want to announce any specific dates but we're making good progress I want to say thank you to some awesome
00:38:07.400
people so M papis was the one who helped me initially with all the SM stuff getting it to a reliable place so um
00:38:14.359
really big thanks Wayne helped me with uh some SM stuff that mapus was not sure about so SM is definitely not a uh a
00:38:22.240
thing that regular mere mortals want to be using but I think I think it it provides a lot of value for our use case
00:38:27.640
um Austin Bales did all the mockups that you saw and um it's driving basically the the implementation Patrick who works
00:38:33.560
with me at Tilda um is working on the the Coco app um and Terren front front
00:38:40.359
row uh he he's been doing a lot of testing so something I didn't talk about is Terren has actually been taking the
00:38:46.079
static build bringing it to rails girls not without the UI and just seeing if the static build actually holds up to a
00:38:51.520
tutorial and uh we got really really good feedback um I think there were a couple of issues that we hit early on
00:38:56.800
with zsh or something but by and large it was a massively improved experience versus the just follow these five steps
00:39:03.560
for people who are just starting um so I that's really great and um Terrence thank you for that so uh thank you very
00:39:19.079
much and I I actually want to close by saying one thing which is when I started I think it's there's a tendency to look
00:39:25.000
at this tool and say this is mostly a tool for noobs it mostly solves a paino that I don't have um and it is a a very
00:39:31.880
important goal for me for at least me to use it I want to use it so I'm we're not really making trade-offs where if you're
00:39:38.480
a noob it's going to be great but then oh I want to build real stuff with this this is impossible like um and I think
00:39:44.240
dhh says this right which is um everybody has to cross a barrier between being a noob and being an advanced user
00:39:50.280
and if suddenly they fall off a cliff because the tool stops working that's terrible so you should build tools that are good for real developers and they
00:39:56.359
and if you do a good job happen to work well for NOS um so that that's my philosophy on the project thank you very
