00:00:17.080
so we're on at the end of Ruby on the last talk
00:00:22.359
uh Ruby is pretty awesome so far I hope I don't ruin it for you guys
00:00:28.640
uh so I'm here to kind of tell a story um
00:00:33.800
about the last uh about year can you guys hear me how far back do I have to be is that good closer ah all right
00:00:42.320
louder all right so the story is kind of about the last uh year and how um we
00:00:50.160
went from not really knowing much about sandboxing to knowing more about it and
00:00:56.039
I'm going to kind of bring you guys through that and go through some of the um lessons we learned and some of the
00:01:02.440
tools that we're using now so this is a picture of my dog her
00:01:10.000
name is Piper um I put it up here just so it would relax me a little
00:01:20.240
bit all right so uh last year a little over a year
00:01:26.360
ago um I me and Greg Pollock were
00:01:31.560
interested in doing a tutorial session at lonar Ruby conf and um we've had some
00:01:38.759
we had some problems before in the past um trying to get everyone's machine set up I mean it took you know like an hour
00:01:45.880
to sometimes people didn't have xcode installed and then they couldn't get Ruby installed and databases and all
00:01:52.439
these different problems so we kind of wanted to nip that in the bud and obviously we were huge fans of uh Tri
00:01:59.520
Ruby so uh I started uh getting the idea that maybe maybe we can do try rails you know
00:02:05.799
so do rails in the browser uh and so this was kind of the
00:02:11.440
first iteration of that uh that's my design I'm not a designer but anyways it worked um you
00:02:19.480
could submit code um to the back end and that that backend would uh execute it
00:02:26.280
and uh it would tell you if you're correct or or not so we had like a bunch
00:02:31.440
of challenges and and uh it worked really well at the conference we had about 30 people come and uh you know we
00:02:39.239
started the tutorial session and right away we were doing work you know we were teaching people um no hassles uh well a
00:02:46.840
couple couple conference Wi-Fi problems um I'm sure you guys are familiar with
00:02:52.599
that um but it worked really well and we got some really really great feedback um
00:02:59.080
everyone really liked it and and a lot of there was a lot of super beginning people never seen rails before and it
00:03:05.920
was really cool to see them just get right in and start writing rails code it it was pretty awesome so um we did a
00:03:13.840
kind of a dumb thing and that is we decided to um release it to the
00:03:19.280
internet um we made it pretty you know we had a designer go through it and and
00:03:25.560
do some designning things and uh you know tried to clean up some of the coat we made some videos and then we uh
00:03:33.120
released it to the world and um people really like zombies
00:03:40.360
um so everyone kind of started talking about it U meta filter um I don't know
00:03:46.159
if you guys do meta filter but um Hacker News Reddit Etc and pretty
00:03:55.360
immediately uh servers went down
00:04:05.360
and uh at first we thought okay of course you know it's rails so it can't
00:04:10.879
scale but turned out it wasn't really a scaling
00:04:16.000
problem this is from uh meta filter
00:04:21.479
thread yeah uh so that was nice of Zed shot pop in
00:04:27.639
and ruin our day um but no it was all over the place I mean anyone with a
00:04:32.759
little bit of knowledge could have broken it right uh we got a bunch of oops got a bunch of bad code coming
00:04:40.320
in uh all over the place um some some just weird code
00:04:47.520
uh had nothing to do with the lesson we were teaching just just some weird I don't know I went through the datab base
00:04:53.919
and looked through like all the code submissions and that was a weird one I found so I figured I'd show Do It
00:05:01.199
um luckily we were on Heroku um thanks to them we you know
00:05:08.479
could easily sort of restart the servers and and you know they have their their
00:05:13.840
whole like uh chroot jails and stuff so no one can really take take uh you know
00:05:19.560
take out actual physical boxes and stuff um also we're very
00:05:25.880
sorry so this was the initial code I wrote to
00:05:32.000
try to block people from doing bad things there are no exploits in this
00:05:40.720
code no there's like a million exploits in this code I didn't really know what I was
00:05:48.680
doing um so quickly obviously we realized that this this wasn't the way
00:05:53.759
to go um and we started looking into better different ways of doing it um we
00:05:59.680
got some capable smart people on our team to work on it um and so sort of the
00:06:04.960
the rest of this talk uh we be talking about a couple of ways we've been doing it better um then after that uh we're
00:06:12.800
actually going to have a live sandbox breaking competition um that you guys can participate in hopefully um and the
00:06:20.520
reason I'm doing that is because whenever people talk about sandboxes um programmers feel like they
00:06:26.400
need to break them so I feel like that's uh perfect um for this crowd and then I'm
00:06:33.039
going to talk about some conclusions of what we found and maybe a way of going
00:06:38.680
forward so um to start let's kind of step back and um ask what is actually
00:06:45.080
what are we trying to do with sandboxing right um so really it comes down to uh
00:06:50.840
we want to run untrusted code so we get code from uh people on the internet and
00:06:56.199
you can't trust uh people on the internet so um you actually want to run it and not
00:07:02.919
have it um screw everything up right so uh there's kind of like four
00:07:09.720
things that um that you want to do when you run un trusted code you WR you want to protect against the name SP namespace
00:07:16.319
pollution which basically means um they can't override you know the Ruby worlds
00:07:22.160
like all the other objects and classes and stuff like that um so you want to kind of like um isolate uh each person
00:07:30.319
submitting code from everyone else also you want to block D dangerous
00:07:36.240
operations from happening so you don't want them to Shell out and erase the file
00:07:41.800
system um you want to limit the resource utilization so basically you don't want them to you know run forever and tie up
00:07:48.280
the process and and uh or maybe do like crazy IO or something like that um you also want to protect secrets
00:07:56.440
so uh you don't want them to really I mean if they if you don't want them to look at at anything on your server
00:08:03.479
probably right um so those are kind of like the four criteria that um
00:08:09.039
sandboxing falls into uh there's another one though that I think is kind of more
00:08:16.039
important oh that's a boring slide that's what that's supposed to say all right you're supposed to do something
00:08:21.479
fun right with the unrusted code that you get um you want to actually allow
00:08:27.720
people to use uh use Ruby right uh because you can lock you could lock Ruby down I wrote
00:08:35.360
this this afternoon um basically this only allows you to you know create an
00:08:40.519
object or add integers together so I mean this is pretty pretty uh sandbox
00:08:46.080
you can't really do anything but it's not that much fun right all you're doing is adding numbers
00:08:58.200
together so that's no
00:09:03.920
good so there's uh with sandboxes this sort of a tradeoff
00:09:10.079
um you basically there's a well in this scientific graph there's a curve that's
00:09:16.160
straight uh goes from being uh more safe uh the safety of your running unrusted
00:09:21.640
code to more fun and those are kind of um inverse of each other so uh
00:09:29.959
really the goal is to get as far down into the fun area as you can like on the
00:09:35.839
x axis I guess um and actually when I was looking at this uh when I first made it it kind of reminded me of of drugs
00:09:42.920
right cuz like the more fun drugs are the less
00:09:49.240
safe so really uh sandboxing Ruby we basically want to give them more drugs
00:09:55.360
right uh or in this case more Ruby which is the same thing thing kind
00:10:01.440
of um and and and these are I pulled a couple of the reviews that we got for zombies um and and people say like we
00:10:10.600
have thousands of them and a lot of people use the word fun they they say fun all the time um they have fun doing
00:10:18.279
this and I kind of want to just th show a couple of these show that like you know it's important that we kind of get
00:10:24.640
this right and not limit what you can do too much so this guy had he had fun learning
00:10:31.480
gave us five stars um this guy had a great introduction and for some reason
00:10:37.079
gave us one star I'm not sure about that all right so from the beginning
00:10:42.160
with some other techniques um of course there's um the safe Global uh has anyone
00:10:48.279
ever used the safe Global in production or anything like that yeah um so
00:10:53.519
basically this is a global flag that defaults to zero and that basically
00:10:59.360
means that it just runs and runs anything right um and then there's a couple other levels you can go up and
00:11:05.560
each level you go is is more restricted to what you can run um and this um sort
00:11:11.800
of safe and and the whole like tainted thing go together so when you get a say you get a pram from a rails controller
00:11:19.240
and uh you say it's tainted of course yes it's tainted because it comes from outside of the environment so anything
00:11:24.880
that comes from outside of the environment is automatically tainted um you can untaint things
00:11:32.200
um uh but not when like a certain safety level is set only when it's safety level
00:11:37.279
zero unless you hack around C code I'm pretty sure there was something at one point that could do that uh I forget the
00:11:43.279
name of the project but anyways um so if you have safe level zero any kind of
00:11:48.920
Tainted code can do anything um so say you set it to one uh codes tainted and
00:11:54.440
you eval it and it's trying to do like a system call so that it doesn't allow that to uh to happen um this actually
00:12:03.000
was written up in like the original uh prag POG prag Prague book uh Ruby book
00:12:10.040
uh and actually I think it has like sandbo it uses sandboxing and Nam spacing and words like that
00:12:16.399
um but another um level is or
00:12:22.800
sorry um another way that's commonly used is with threads so you create a um
00:12:28.320
a thread and you set the safety level in that thread only um and then the parent
00:12:34.120
environment basically doesn't doesn't uh take the safety it's still safe zero but
00:12:39.279
the the thread has safe three um and this is kind of how it's written up in the book and um it's
00:12:47.720
pretty useful but it's kind of like a blunt sword or I don't know what's another analogy for that but it's it's
00:12:54.440
kind of a uh someone help me out with that that one yes it's dumb
00:13:02.240
okay um and like for instance when we originally tried to use this on Rails for zombies um shock or rails doesn't
00:13:08.920
work if it's in a safety level um so you actually couldn't uh couldn't do anything with it um so we couldn't
00:13:15.480
really go this route unfortunately uh I think for maybe maybe like very simple things you could
00:13:21.480
probably use safe but uh considering this is what y wrote back in like
00:13:26.519
2008 um a lot by then no one really used safe so I you know I
00:13:31.920
doubt anyone's going to use it now um so that's that's
00:13:41.000
safe so um another way we tried doing it was with uh semantic analysis and who's
00:13:47.440
in that laser talk previously that was insane
00:13:52.519
yeah um we we're not really doing anything like that um this is a I guess that's that's static analysis this is a
00:13:58.600
little a little bit different semantic analysis um so this is kind of like it's kind of like that gsub thing
00:14:05.839
but on steroids um so it uses the uh uses Ripper which um hopefully will be
00:14:13.480
complete um with those bug fix fixes from the laser guy pretty soon but uh
00:14:18.839
it's pretty good so we use basically you parse the Ruby or the Ruby code string
00:14:23.959
that you want to run um and then you uh walk B basically the U output of that
00:14:31.160
and try to determine if if they're doing anything bad
00:14:37.399
um and we built something called Ruby cop um that's kind of like does a lot of
00:14:43.639
the hard work for you right um so it basically you can like create a policy
00:14:49.959
that defines like what things are allowed what things aren't allowed kind of thing um you can Whit list constants
00:14:56.279
say or Black List constants and and uh and and and then run uh run the code or
00:15:02.600
not actually running it but pass the code to like the node Builder and find out if it's acceptable or not um so yeah
00:15:09.519
we use like blacklists for calls so we're sort of like you can't call these
00:15:15.560
things right um because they could possibly be dangerous um same thing with
00:15:20.920
the constants um same kind of deal there um so this is this isn't really a Sandbox
00:15:27.959
right but it sort of meets some of our criteria of sandboxes so
00:15:34.839
um one good thing about it that other sandboxes don't really do is that you're really only sandboxing the code that um
00:15:42.319
was submitted to you and and not the code that you already have running like in your Ruby process so uh one big
00:15:49.000
problem with the safe thing like I was talking about was that like if you turn on safe um for that thread and then you
00:15:55.399
run someone's code that calls something in rails well down in rails does something that's unsafe and now and you
00:16:02.240
can't use that right cuz like something in rails should be able to do that but the code um that was submitted should
00:16:08.959
not so this is kind of a good way to get around that problem uh there are a lot
00:16:14.040
of Cons with this though um it doesn't really create a new isolated environment right it's not doing anything with the
00:16:19.639
environment it's just looking at the string of code that you submitted and making sure it's not going to do anything bad
00:16:26.279
basically um if the code gets through can do anything and it's too restricted right
00:16:32.920
it's kind of more in the G sub variety where um you're not really giving giving people the full power so they can't use
00:16:40.120
like like we block send right so if if you call send that's we don't run it so
00:16:46.160
really you should be able to do that and that that's kind of a bummer right oh and there are a lot of lot of
00:16:53.040
Ruby edge cases
00:17:04.839
obviously this is just a small sample but these are some of the weirder ones um that we found I'm sure like Charles
00:17:12.360
you probably know a lot more um but um surprisingly we have done
00:17:18.760
like 1.3 million code submissions um using this and we don't really ever go down uh
00:17:26.839
probably until after this talk cuz you guys are going to go break it basically
00:17:32.120
um so even though it's not perfect um and it kind of limits a lot of it it it does it does work um for our
00:17:40.480
uses um and that's sort of like gotten us this far um but we we kind of need a
00:17:45.960
better way to do it so recently um we
00:17:51.160
couple months ago I uh took over I sort of took over the TR Ruby uh project and
00:17:58.280
uh I decided to rewrite it using our like code School engine and all that but I I wanted to use a better
00:18:04.159
sandbox um so we uh we kind of obviously used to try Ruby written by why so we
00:18:10.919
figured why not like kind of rewrite the freaky freaky sandbox and um luckily it sort of had
00:18:17.919
already be been like Rewritten a couple times and uh we found one that was
00:18:23.440
Rewritten in J Ruby called Java sand um but it was um I think it was targeting one two or 13 and it was it wasn't
00:18:30.559
working anymore so basically we stole a lot of code from that um updated it for the new 16 J Ruby stuff and uh
00:18:38.400
implemented this sandbox so it does a lot of things that we want it uh isolates the name space um so this is
00:18:45.559
just a a plain sandbox uh eval some code and then uh you know you can call that
00:18:51.559
code inside the sandbox but you you don't have it outside the sandbox so Fu does not exist um outside the sand
00:18:58.320
sandbox only inside the sandbox uh and we kind of do that um with a great thing in J Ruby um where it
00:19:06.000
gives you kind of an API to create a new instance of Ruby I wish every um Ruby
00:19:12.240
had this this would be awesome um but only J Ruby I don't know about I'm sure rubinus has something like this but um J
00:19:19.840
Ruby had it so we rewrote this in J
00:19:27.240
Ruby all right also blocks uh dangerous operations so um I like I showed before
00:19:37.360
I just did sandbox. new so that that's just uh it doesn't actually uh it just isolates the name space but sandbox.
00:19:44.480
safe um once you activate it will actually uh un like remove a bunch of methods um that are dangerous uh like
00:19:52.240
like the system call and all that kind of thing so it blocks dangerous operations from inside the sandbox um it
00:19:58.880
also we integrated like fake FS so when you're running a safe sandbox um you
00:20:05.760
know you can't read and write from the actual file system but you can um from fake FS which actually TR Ruby uses
00:20:12.400
because it does has some uh lessons on file stuff so that was pretty
00:20:18.000
handy um it also uh limits resource utilization so when you call eval you
00:20:24.159
can pass timeout and uh that's in second obviously and then if it if it runs
00:20:30.080
longer than that um it'll time out and it won't it won't you don't block your whole server uh is doing it in a kind of
00:20:36.760
a nasty way right now with u creating a a new thread calling joint on it with the time um and then calling kill on the
00:20:45.000
thread uh kill bang which I think is the original way that why did it um and he
00:20:50.080
said it was the hack thing in there and yeah that's pretty much the heck thing in there so um would like to to do
00:20:57.200
something a little bit better than that but right now I don't think there is anything um also another really cool
00:21:03.080
thing with um this sandbox is you can give the sandbox capabilities so you can
00:21:08.240
define a class or or whatever outside of the sandbox and then uh reference it
00:21:13.320
inside the sandbox uh and then it does like a it communicates with like the outside class to call methods using like
00:21:19.880
method missing and stuff and references um hidden inside the the J Ruby
00:21:25.440
runtime um so this is really cool for especially for like TR Ruby stuff where
00:21:31.240
uh we you know we write up some sort of like popup um library that they're supposed to use so we give a reference
00:21:37.360
to the popup Library um and you can like return objects back and forth between the sandbox and and um and the native uh
00:21:46.279
environment and uh it uses like Marshall dump and load and that kind of thing um
00:21:51.320
so yeah like I said before uh we use this to build uh TR Ruby uh which is up
00:21:57.000
now and I don't um it hasn't really been hacked
00:22:02.320
yet um you guys can go ahead and and try to hack it
00:22:07.559
but um Instead try to break this thing that I wrote um it's called a Sandbox
00:22:14.240
breaker 3000 it's written just so people can try to break these
00:22:19.960
sandboxes let me show you that
00:22:25.240
quick o that's a small all right um so if you go to that
00:22:31.919
URL U and then hit new submission yeah if you go to that URL hit new submission
00:22:38.360
and then uh you'll be taken to I'll show you
00:22:44.799
there taking this screen here um submit your put your code in there uh if you
00:22:49.880
want to put your name whatever you can put your name um then choose which sandbox are so like I just kind of
00:22:55.440
described the jail one is the freaky freaky uh J Ruby one um cuz that's kind
00:23:00.919
of what they're called they're known as like jails those those style sandboxes and then the semantic analyzer is Ruby
00:23:08.080
cop um and then when you guys submit code it'll actually show up here live
00:23:14.039
like that so someone just submitted fork and and that doesn't work
00:23:22.760
so but would take a couple minutes just to try to break it and uh I'll walk
00:23:28.360
around we can talk about trying to break it in stuff
00:23:38.159
so there's a little bit after
00:23:46.320
yes can I ask a question sure yeah yeah that works would it be if you this
00:23:52.200
entire thing in a different process in a separate process because if there's there
00:24:02.159
exension yeah so everything that we run like on co- school and here um the actual code gets executed on a separate
00:24:08.480
server than the app server so this is actually when you submit the code it'll it makes an HTP request to the actual
00:24:14.880
executing server and then is that is that kind of what you were asking about
00:24:21.559
Rec oh there's a mic out there you can walk around with too if you want well just
00:24:29.440
sorry about that um repeat the question so the question was uh do you
00:24:35.279
want shouldn't you execute the code on a different box is
00:24:51.960
that uh well that's that's kind of how we're doing it right now um but we're
00:24:58.120
sort of using um like we have many different instances
00:25:04.440
of the executing server right so I guess if they take one down we have other instances that are up and then it'll
00:25:10.520
fire it back up is that kind of what you mean
00:25:26.679
symb
00:25:34.559
yeah it's not bad you don't
00:25:40.279
know break it was that Aaron back
00:25:47.279
there well actually really it's impossible to actually break this out at
00:25:52.360
least I thought so yeah
00:26:00.720
look at the Heroku and see what
00:26:23.200
happened I was just
00:26:28.279
yeah that's a tough
00:26:33.399
one yesterday I did an attack on ffs where I uh basically opened a file and
00:26:39.520
just kept writing data into it until I took all the memory just
00:26:50.320
F it's recorded
00:26:56.600
right which one were you was that the jail one is that that's broken
00:27:19.120
SS yeah so I think what happened there was in the logs there was a post error
00:27:24.320
cuz the code submission was too long for the just cuz it saves like the out
00:27:32.520
totally I'm so this one looks the Strong Bad
00:27:37.720
looks like yeah that's a big problem uh the whole xxs attack huh yeah
00:27:45.600
it didn't work when I gave this presentation at work they they they broke the um xss uh vector and we're
00:27:55.159
putting on funny images of dogs and such but fix
00:28:09.760
that
00:28:19.440
H hey still run that's not bad I think the semantic one's probably
00:28:26.440
uh it's probably easier to break but
00:28:33.519
uh sure
00:28:38.679
have
00:28:54.480
right um no nothing like that really yeah these are um deployed on
00:29:01.440
like heru nothing really that special so although Heroku really does a lot a
00:29:08.600
lot of stuff free for us um server wise like we actually deployed these who who
00:29:15.559
here works at
00:29:20.600
Heroku don't tell um but they use like Linux
00:29:25.840
containers right is that how you go do the the virtualization stuff yeah which
00:29:31.559
is um sort of something that I've been looking into although you know for who does it so it's kind of like we get for
00:29:37.919
free but it um it doesn't allow people to do like really malicious stuff um if
00:29:43.799
they get an exploit um the worst thing they can do is take down our app um
00:29:49.519
which isn't too bad right and that's there could be a lot worse stuff than
00:29:55.120
that GC stress that's a new
00:30:13.039
one is that what well all you guys are like isolated
00:30:18.279
or each request you guys do is isolated so like if you do that um should isolate it from the rest of the
00:30:26.240
requests yeah do it and then do BU of bad
00:30:34.760
stuff so I'm guessing you're like you have I'm guessing you know like a bunch
00:30:41.919
of exploits right off the top of your head on this kind stuff there probably a few depending onion of
00:30:50.519
JB a while back that' be an easy one to do just it's a particular big decimal
00:30:57.279
you try and create it really just to improve the
00:31:14.679
would catch it and be like all right that's a question um so if if the
00:31:19.720
timeout it's running and it's in for example a piece of code like that that's
00:31:25.600
written in J Ruby WR and we don't we're not checking red kill
00:31:32.080
in case okay do you do anything for that at this point no because the for example the the
00:31:40.240
Java thread uh there are the unsafe apis that you can kill or suspend threads
00:31:47.159
that you're never supposed to use but in this sort of case you could actually just physically kill theat that point at
00:31:53.559
J oh in the jva kill the threat right not from so you can get a reference to the the Ruby thread the actual Java
00:32:00.799
thread that goes with the Ruby thread and then have a raise I got you this is actually
00:32:10.080
somethings so we is right
00:32:17.320
now and and running tests against
00:32:26.080
we so we didn't realize that this point now guess like
00:32:33.080
guys don't what yeah that was a good
00:32:43.799
one that one right there hey it's still
00:32:51.320
running just took out an executive yeah after the exception everything like this so
00:32:59.000
they will continue to track run yeah time out does work pretty well
00:33:06.799
all right well um this will be up I guess if you guys want to keep hacking on it
00:33:13.120
um and
00:33:18.480
the oh oh yeah all right so kind of wanted
00:33:24.600
to propose um what I'm calling the freaky theory of
00:33:30.200
sandboxes um and this is sort of based off of this guy Alan Ginsburg he was a
00:33:35.880
be poet um and he came up he wrote how if you guys know that um he came up with
00:33:43.919
a um theor oh the whole build thing is off oops and he based it off of the
00:33:52.080
um the laws of thermodynamics as applied to someone who's playing a game so he
00:33:57.240
said uh number one is you can't win uh which relates to the conservation of
00:34:03.200
mass and energy um you can't break even uh which relates to uh law number two
00:34:08.520
which is entropy increases and you can't quit the game uh which relates to law
00:34:13.599
number three uh the impossibility of reaching absolute zero so I'm basing my freaky theory on his
00:34:20.240
theorem and that is number one there will always be another exploit number two one exploit is enough to take
00:34:26.599
everything down um but no amount of exploits to keep us from trying to get this right um
00:34:34.480
and and making people happy introducing people to Ruby in an easy and fun
00:34:40.119
way um yeah told you that was
00:34:46.839
on all right so uh you know I'm hoping to sort of like get a little bit more
00:34:52.200
interest in people maybe implementers maybe not um interested in and um making
00:34:58.200
it so we have like a better sandbox for Ruby uh we're trying to do that at MV
00:35:03.720
labs and for code school um you can find our uh the code for the Ruby cop which
00:35:10.240
is the semantic analyzer and the uh J Ruby sandbox which is the the free
00:35:15.320
version uh of there uh you can keep pcking on the uh sandbox breaker if you
00:35:21.040
want um and in conclusion I want to thank Dre Lacy who who um did write a
00:35:28.680
lot of this code um but couldn't make it to Ruby conf and and didn't want to do a presentation on it so I sort of had to
00:35:36.440
um so thanks to him and thanks uh for coming out and have a good