00:00:17.279
so uh in the program this talk is titled Next Level virtual machine maneuver uh I
00:00:23.199
emailed the organizers a few weeks ago um but the programs were already printed out so there was a slight topic change
00:00:29.599
but it's still pretty close um so instead of Next Level virtual machine maneuver it's uh actually next level
00:00:35.719
testing which is kind of a level up from this um as I was writing the previous
00:00:42.000
talk I realized it's kind of more of a very specific like here's how you do this and here's how you do this and
00:00:48.039
you're all pretty smart so you could learn that in a blog post um when I go to conferences I like hearing more um
00:00:54.160
about ideas as well as things I could take away uh and use at work so I'm hoping this talk delivers both of those
00:01:01.920
things so very quickly about me um my name is Mitchell ashimoto this is my
00:01:07.600
gravitar Twitter handle and GitHub URL the best places to find me online um
00:01:14.600
it's not really important right now but I am one of the creators of vagrant if you use
00:01:20.960
that I work at a company called keep and we do mobile advertising that looks a
00:01:26.200
little bit like this and I wanted to thank engine yard for um helping me with
00:01:32.000
my work on vagrant and also getting me out here to talk so in 2010 at
00:01:40.799
Goo uh rain you might recognize him from the panel rain gave a talk called real
00:01:46.119
world testing and he did a really good job of kind of explaining the landscape
00:01:51.399
of test Frameworks within the Ruby World um you know covering rspec cucumber test
00:01:57.280
unit all all those things as well as talking about about what tdd is what bdd is and so on it's um and it's kind of a
00:02:06.399
entrylevel view into testing and if if you want to learn more about that you should really watch this talk on confir
00:02:12.200
but my talk is kind of the next step or kind of a sequel to this talk I don't
00:02:17.239
cover beginner topics at all I'm going to assume that you all know how to use test unit um and I'm going to dive right
00:02:25.640
into things you might see in a larger code base or you might run into in in real projects after testing for a
00:02:33.640
while so let's start talking about testing yeah uh but before we get
00:02:40.040
started we're going to I promise like the first 10 15 minutes is going to be kind of abstract and academic and then
00:02:46.640
we're going to get into all concrete stuff you can use every day but I think it's important to to think a little bit
00:02:54.640
differently about testing uh especially when you start getting into more complicated scenarios
00:03:01.200
so the question that I want you guys to kind of think about is is why do we write tests uh I know at least in my
00:03:08.159
case I was taught from very early on that you just write you just you just do it and it's a good idea because you want
00:03:13.200
to make sure your code runs um uh but only recently I've started wondering like why am I writing this test
00:03:20.239
specifically what purpose uh does testing have so one of the most common
00:03:27.040
answers I get or the top answer I get is stability and it's totally a good answer um you
00:03:33.920
want to make sure your code runs you want to make sure it works so you write tests for it uh but if you if you're
00:03:39.640
agreeing with this then just I want you to think a little bit deeper and think why why do we care about
00:03:45.319
stability another answer I get perhaps is performance our tests are a very quick way to verify that some methods
00:03:52.079
are taking a very long time if you might have performance tests or you might just see like that one dot is taking a little bit too long to show up um and again
00:03:59.680
again this is a good answer um and again I want you to think one level higher and think why is this important at
00:04:06.920
all uh and the third most popular answer I usually get is to test bug fixes to
00:04:12.720
verify we don't get regressions in our code uh this is kind of hand inhand with stability um and once again why does
00:04:20.280
this matter so we have these three things they're all very good answers to why we test but I think there's a more
00:04:29.080
General reason overall to why tests are good
00:04:35.199
so my what I what I came up with or why I think that we write tests is to assert
00:04:42.600
there's a certain value in our code um because as much as we care about code
00:04:49.280
Beauty and making sure it's very modular and and easy to extend and all that our our code is worthless The Code by itself
00:04:57.000
is not why we're getting paid we're not being paid to to type on a keyboard and write code the business people the suits
00:05:02.680
your managers they don't care about the code at all and especially end users when I use a website when I use you know
00:05:09.440
Twitter when I use Facebook I don't care what is going on behind the scenes I just want to see something work so what
00:05:16.160
is money is the business value that your code is giving
00:05:21.240
out what what everyone sees is what your code does and not how your code's written um so we're really writing tests
00:05:30.000
in a way to assert that we're getting that value that we expect out of our tests and that's kind of abstract but
00:05:37.960
we're going to go a little bit little bit further um there's a study in Academia called information Theory it's
00:05:43.319
a very very broad topic um I don't think anyone would describe thems as an
00:05:48.440
information theorist probably something a subfield within that uh some things information Theory well what information
00:05:55.520
theory is is is the quantification of information treating information as a quantifiable value and
00:06:04.280
some concrete subfields are data compression um a lot of electrical engineering signal
00:06:10.759
processing um natural language processing uh these are all very important things but but the abstract
00:06:17.080
notion of information theory is is very very interesting so in 1928 a professor named RL Harley
00:06:25.240
published a paper um I think actually I have the URL down there yeah it's titled trans transmission of information and he
00:06:32.199
made a groundbreaking in terms of information Theory a groundbreaking idea and the idea it's a little bit dense so
00:06:38.960
I'll go through it a few times um the idea was that information could be
00:06:44.000
Quantified as a log function of the probability of that of an event
00:06:49.960
occurring uh so I'll just say one more time so you could think of the parts but information could be Quantified by the
00:06:56.560
log function of the probability of that event occur occurring um so in more layman's terms
00:07:03.360
if something has a high probability of happening we don't learn that much from it because we see it all the time and if
00:07:09.080
something has a low probability of happening we're going to get a lot of value from it because you don't see it that often so a lot more things are
00:07:14.199
going to be new to you and when you put it that way you kind of think well
00:07:20.000
that's kind of common sense that that makes sense and it does but and in
00:07:26.879
general I think people say this makes sense but it's funny because when it comes to testing uh when I work with a
00:07:33.199
lot of developers when I I was a consultant for 3 years so I saw a lot of different teams um I don't see a lot of
00:07:39.400
common sense when it comes to testing it's like we forget this idea when we test I I I see a lot of unnecessary time
00:07:48.039
uh and money spent on writing on going for 100% test coverage when you're
00:07:54.080
testing things that are not giving you that much value in return
00:08:00.360
and so the question The Guiding question that especially once your test we gets larger uh when the stakes are higher
00:08:06.560
that you need to think about it's how much value are you getting out of your tests and if you could quantify that
00:08:12.720
then you could say this test just isn't worth writing right now and this test which will maybe test this thing I
00:08:19.800
should do that instead so we need to test properly and in small in small libraries
00:08:27.400
when your codebase is small um it's easy to to not see the issues that
00:08:33.800
come from from wasting time on tests but as your test we gets larger as you have
00:08:39.760
more to do as you become a busier person it's very important to optimize and think about your
00:08:45.279
tests um a good couple good points to stand by is if you're succeeding too
00:08:50.959
much too much if your tests are passing too much it's probably not because you're some uh you
00:08:57.079
know great holy program prammer uh because we all make mistakes it's probably just because you're testing the
00:09:02.240
obvious things you're testing things that you're expecting are going to pass uh and and of course we don't know
00:09:08.440
what's not going to pass so one way to do that is to move from say a unit test level to an integration test level or
00:09:14.680
vice versa if you're suceeding too much at one level move to another another level of
00:09:20.680
testing and another important point is We Gather the most information through optimizing both failure and success uh
00:09:28.920
if if you graph out Professor Harley's uh function you get this upside down
00:09:34.320
Parabola and where the x-axis is probability of something occurring in the Y AIS is the value that you extract
00:09:42.000
um and so the maximum value you get is at 0. five probability if you don't if
00:09:48.120
you if you think you're going to succeed as much as you think you're going to fail you actually have no idea what's going on at all and you're going to
00:09:54.160
learn the most out of everything and that's what you attempt to achieve through tests
00:10:01.720
so okay so that was the the academic floaty 10 minutes of the talk um so now
00:10:09.800
let's talk about real concrete testing and and how we could how we see value uh
00:10:14.959
what value these tests are giving us but more importantly let's not talk about exactly how you write each test but
00:10:21.279
let's talk about the ideas behind the tests why you would choose one over the other and um and the benefits of each
00:10:29.160
how all these work together so what I'm going to be introducing aren't new ideas I don't at
00:10:35.680
all claim to be smart enough to come up with these on my own these are ideas that I've seen from other communities I
00:10:41.959
am a full-time python developer so I do that and then I also do a lot of jvm work so I see Java stuff um so these
00:10:49.160
ideas come from java python uh my work experience and
00:10:55.160
personal experience so by personal experience I mean open source and
00:11:00.639
um so let me quickly give a couple qualifications of why I'm even qualified
00:11:06.120
to talk about this subject uh so you don't so at least you maybe you could believe me a little bit uh my first
00:11:12.839
qualification is vagrant my open source project it's very quickly is a tool
00:11:18.399
which manages the Automation and life cycle state of virtual machines for the
00:11:24.040
purpose of development and it is used at least in part
00:11:29.720
uh by all the companies listed up here and many more like Tumblr and Eventbrite
00:11:35.880
uh and Barack Obama use it for their entire website and then Adobe and living social use it for parts of the
00:11:40.920
development but this is just a subset and some hurdles that I have with
00:11:47.120
vagrant is it has huge external dependencies it uses virtual box as it virtualization layer currently and if my
00:11:55.040
communication with virtual box breaks down then the whole project doesn't work so I need to be able to test that unit
00:12:01.200
tests aren't enough for me but the integration tests are extremely slow uh creating virtual
00:12:07.720
machine one virtual machine acceptable creating hundreds of virtual machines for the purpose of testing extremely
00:12:13.880
slow uh I have around 50 integration tests right now and they take around four hours to
00:12:19.360
run there's a lot of os specific edge cases uh that I need to again I'd like
00:12:25.399
to at least verify uh for example in for some reason on OSX 10.5 and earlier the exact system
00:12:32.839
call doesn't work very well with Ruby so I have a work around for it and when I
00:12:38.279
push a release I want to make sure that the code that invokes that is continuing to
00:12:44.279
work and vagrance made to run in any modern environment so it should run on all your laptops it's designed to run on
00:12:51.040
all your laptops and I only really work on a Mac so again when I push a release
00:12:57.320
it would be nice to know that that vagrant Works in every environment that I intend it to work
00:13:02.639
on and U more recently there's a higher cost for mistake um if I make a mistake
00:13:08.199
and I push a broken release it might damage my credibility that's less important to the more important fact
00:13:14.199
that it could damage the reputation of the project itself um a lot of people only recently these big companies have
00:13:19.959
been moving on vagrant because they've seen it work for the past two years and if I push something that totally breaks
00:13:25.360
their whole development team and causes them to be unproductive for a day then they will never touch it
00:13:32.320
again and my second qualification is I don't know why it says credentials but my second qualification is my work keep
00:13:39.880
uh like I said we do mobile advertising so we're mostly an API shop uh we don't see a lot of traffic but 500 million API
00:13:47.160
calls a month it's a okay amount uh there is real money at stake in this case for mistakes uh there's
00:13:54.880
real credibility important credibility especially as a startup at stake and there's also potential legal trouble
00:14:01.320
like we have some contracts where for example we're brand partners with Carl Jr and we're developer partners with an
00:14:07.600
exercise app and it's in our contract that if we show a Carl Jr ad in an exercise application then we're liable
00:14:14.120
to be sued for up to like $500,000 so we need to know we need to know when we push code that our
00:14:20.480
algorithm will not show that ad in in certain certain
00:14:25.720
games so at this point with all the these things with all these you know all
00:14:31.320
these ideas we need to do a little bit better I realized in my testing I need to think about things a little bit
00:14:37.440
differently because there's a lot of things to test and I want to test all the things uh but I want to do it
00:14:42.639
efficiently and I want to do it uh correctly so I'm going to Qui quickly
00:14:47.959
shotgun out the ideas I'm going to talk about so you could quickly see the list and then I'm going to go over each one in detail
00:14:54.399
so I think that I think that's it these are these are what I'm going to talk about today
00:15:03.079
okay so the first thing I'm going to talk about is integration testing um I
00:15:08.320
love integration testing it's it's what I do most nowadays so when I say
00:15:14.440
integration testing I mean blackbox testing so here's a real valid integration test so I Echo
00:15:23.040
something into Foo and I cat Foo and my my test here in this case would assert that I get hello back because that's
00:15:29.040
what I put into the file and this is a valid integration test of cat I don't
00:15:34.199
care how cat works I don't care how fast it works that's not the purpose of the test I just want to know that given some
00:15:40.279
state of the world when we start I get this state of the world when we're
00:15:45.399
done and the purpose of integration test is you test against Real external dependencies so this is if you don't do
00:15:52.800
this then the whole test is kind of useless you need to make real networking calls uh real file creation
00:15:59.240
if you can get it you should use real data and there should be no access to internal code the integration test
00:16:05.720
should not be able to access internal libraries at all um and this is all very
00:16:12.160
important and in the case of real data what I mean by that is for example for the selenium tests for uh my work we
00:16:18.519
have a script that pulls in a subset of production data anonymizes it and uses
00:16:25.199
that um integration tests are the best way to test platform specific edge cases
00:16:30.680
and also to find platform specific edge cases uh because they are running through your application and your
00:16:35.880
website uh in ways that the end user would use them so they find the plat the edge cases that you actually should care
00:16:43.399
about and the best way to do this is virtual machines and the cheapest way to get it going is to use vagrant U because
00:16:50.480
you could run multiple os's on your own machine and run the tests in there um as
00:16:55.600
you get up and running more CI not that hard anymore and pretty inexpensive as
00:17:01.639
long as if your project is big enough to to want to test so many platforms then
00:17:07.360
CI is relatively cheap and integration tests are the best way to get the biggest bang for your
00:17:14.319
buck uh usually when you start a library uh so do it early that's important so
00:17:20.919
when you start a library you know what that Library you how you want it to work so later today Tom Preston Mor is going
00:17:27.520
to talk about read me development and that's usually because you know how you want it to work so you could write that read me first um likewise you could
00:17:34.679
write tests asserting how you want it to work the whole thing and that'll start verifying it very quickly um and no
00:17:42.280
matter how good you think you are or how how simple the library is and you think you've got it going uh the internals are
00:17:49.120
always going to change every internal Library when you start or internal class when you start a project is going to
00:17:55.159
change at some point and if the first thing you do you start writing unit tests you're going to have to rewrite
00:18:00.799
all those unit tests and this is where that value thing comes in um when I see a lot of a lot of people that I've
00:18:07.880
worked with just start hammering out unit tests like if I if I start unit test early and I test each method then it just becomes easier for me later um
00:18:14.799
and that's one way to look at it the way I see it is those unit tests aren't really providing much value because your
00:18:20.520
internals are going to change later anyway so why not test the whole thing and your integration test almost
00:18:25.720
never change because you have some purpose in mind and that's what that thing's going to do so write your integration test
00:18:33.919
first uh but I won't lie integration tests are horribly slow compared to unit
00:18:39.679
tests or other sorts of tests uh but don't worry about it I I'll talk about
00:18:44.840
fast tests and slow tests later just write the integration
00:18:50.480
test okay so some people think I just include this because some people think that integration tests are different
00:18:55.799
than regular tests uh and so I I include a real example this is a valid integration test it just uses test unit
00:19:02.360
I only use test unit really and this is an integration test for vagrant that calling vagrant DV will return the
00:19:09.240
proper version so you could see that it's pretty much just the normal unit test except for two helpers there's the
00:19:16.159
execute method that's going to execute it on the command line like a normal user would that Returns the exit code
00:19:21.240
standard out standard error I do some assertions and then remember that integration test cannot access internal
00:19:27.480
code ever so I do pass in a config uh Json which includes some things like the
00:19:33.400
version which I need to know in the test to assert so I assert that the version is equal to that instead of looking in
00:19:39.039
the library of the version because it shouldn't do that breaks the isolation and running it has to be
00:19:45.960
simple uh if you don't make your test simple to run no one's going to run them so I'm a really big fan of just running
00:19:52.000
the file to run the test and that just works with test unit okay so fast tests versus slow
00:20:07.760
tests uh I don't know who said this I don't know if anyone ever said it well I I say it all the time but if you test if
00:20:14.080
you don't run the tests then what purpose do your tests serve at at like I said I was a consultant for three years
00:20:20.039
I've seen probably like 50 uh rails projects when I was doing that and there
00:20:25.840
was I would say over half said look we have an awesome test suite and then I would see all their developers just
00:20:32.200
committing committing committing and I was like what what's the point you don't run the test what's the point
00:20:39.960
so at some point your test Suite will become slow and I don't know why this is
00:20:46.400
in the Ruby Community more than other communities but Ruby is tend to Ideal
00:20:51.760
have this idealized notion that every project you're going to be able to run every commit every test before every
00:20:57.080
commit and you're going to have this awesome like stable on Commit push um and really the pattern that I
00:21:04.799
found is as your project reaches a certain size or if your project is worth
00:21:10.600
anything at all your test Suite is just going to become slow uh you're going to have integration tests those are always
00:21:16.080
going to be slow um you're going to get thousands of unit tests those are going to become slow so you can't expect be
00:21:24.240
expected to run every commit every test before every commit uh so something that I've been doing in practice and at work
00:21:31.200
is establishing a subset of my tests as fast sanity tests these are tests that
00:21:38.520
are run before every commit um they should be very easy to run because you
00:21:44.159
want all your developers to run them and they should contain the a WTF subset of your tests um so another example is
00:21:52.200
sometimes I work with developers and they're like yeah we have this full test weed and we do we we can't run them before every commit but we run these
00:21:57.799
test test because they're they're fast so we just run these ones and and the rest go somewhere else uh I don't think
00:22:04.440
that's enough I I like to get a subset of my tests that at least tests a good
00:22:09.520
chunk of every subsystem or or module of the of the project so that when you do
00:22:15.880
run the test if you have some big what uh almost cussed some big WTF going on
00:22:22.600
then it'll catch it and that's the real purpose of these sanity tests so an example this is how I run my
00:22:29.279
tests I only use mini tests it's pretty easy to do it um so once you have your fast sanity
00:22:34.880
tests you you still need to run your full test suite at some point um and hopefully very often but you shouldn't
00:22:41.960
do it on your own computer because that that's hurting productivity more than helping so you should move the full test Suite onto a CI machine there's really
00:22:49.400
no excuse anymore uh CI is getting really easy compared to a year ago and especially two years ago CI U I'm pretty
00:22:57.520
sure like my little sister who's like 10 could set it up uh Jenkins is like WordPress for
00:23:04.640
CI you just like run it and you get this gooey and you click around and suddenly you're testing um so you should be able
00:23:10.159
to figure that out for Ruby Travis CI is amazing it's very new so if you have
00:23:16.159
some problems that's a disclaimer but it's very cool and you should Google that if you're using a open source R
00:23:22.000
project it's I don't think it's useful for private projects um and if you need
00:23:27.279
a lot more C ization and your kind of Brave build bot is what I use and it's brilliant um the uh comparison I like to
00:23:35.760
make is Jenkins is to Wordpress what buildbot is to rails or some web framework so Jenkins is a full stack
00:23:42.559
solution you just install it and it runs buildbot is more of a CI framework so you have to write code in this case you
00:23:48.919
have to write python code and you build your own CI you have a repository and you kind of build your own CI but with
00:23:55.080
buildbot you could configure everything and they handle all the m Master Slave work distribution all that for
00:24:02.039
you so one caveat to fast sanity tests and to CI and all that stuff is targeted
00:24:09.159
tests if you're fixing a specific bug you should run that test at least or the test that test that bug before the
00:24:15.679
commit I hope this is common sense but I mention it just in case uh and again it
00:24:21.840
should be simple these examples don't really mean anything but they're just to show it should be simple
00:24:29.039
uh there's other options when you have fast tests and slow tests parallelization and test marking and I
00:24:36.840
do talk about both of these so just going to ignore them for now and actually next I think is marks so test
00:24:44.360
marking this is something I've never ever seen any Ruby programmer do uh and
00:24:51.840
they're basically test annotations or test tags which when I first saw them was really gimmicky and I was like I
00:24:59.200
personally I hate tags I've never used tags for any meaningful purpose anywhere except maybe in this case um so here's
00:25:06.279
the example from Python and this is something that I first saw py test and now I try to use all over the place so
00:25:13.720
in Python you'll have a test maybe this is testing a parser and it's testing that it could parse a string that has
00:25:19.320
underscores in it and then python supports first level function decorators so I'm decorating the function with a
00:25:26.120
mark a tag which is issue 29 maybe this test is part of issue 29 and to run it
00:25:32.760
you would do something uh like this this is how you actually run it in Python it
00:25:38.440
kind of fuzzy matches the tags uh so why is this useful this again
00:25:43.559
when I first saw this and maybe a lot of you thinking this I was like that gimmicky not really useful not a power
00:25:49.399
tool uh but it is one of the most important tools I use now in
00:25:55.000
testing it's really useful for for fast test and slow test marking for sanity
00:26:00.159
test marking um you like I I like to keep and most people have most projects I've seen like to keep their tests in a
00:26:05.760
structure that matches their namespace layout or subsystem layout or something like that and that makes total sense so
00:26:12.640
you don't want to have you know test WTF for your your fast test or something
00:26:18.039
using marking you could just Mark some tests like fast or sanity and then when you run them it just runs those tests
00:26:23.640
and the tests still live with the system that they're actually testing and this is
00:26:29.159
great it's very useful for plugins uh I I intended to put an example here but I
00:26:34.520
don't have one but the example that I use it for is I haven't used rails in like a year and a half uh but when I did
00:26:41.440
use rails I remember that on every unit test I think it tore down and created a new database for you or something like
00:26:48.399
that and in practice when I'm testing my models uh I don't need a database an
00:26:54.240
isolated database that often a lot of things I could just test in memory so so at at work what we do is we Mark the
00:27:00.919
tests that do need a database with isolated hyphen DB and when the test Runner sees that it calls something
00:27:08.000
which sets up an isolated database just for that one test and so you could use you could imagine you could use marks
00:27:13.880
for all sorts of things um another thing I'm using it for and this time in vagrant is I'll mark a test with uh OSX
00:27:21.760
10or 5 and that test will only run on Mac 10.5 because it's not testing
00:27:28.600
anything meaningful on any other platform so that's useful uh issue tracking is very cool a
00:27:35.360
lot of projects create separate files like test issues like 825 RB to test
00:27:42.279
like issue 825 and that's very cool it's very easy to find an issue for a
00:27:48.159
specific issue number but now you're moving your tests out of the system they're actually testing into all these
00:27:53.960
files and i' that's one problem I've always had with it with with test marks you could keep the tests in the systems
00:28:00.519
are actually testing and just mark them with the issue that they're working with and that way it's very easy to if you're
00:28:06.360
if you find a regression or something you're like oh this is regression from issue 49 you could just run test issue
00:28:11.519
49 and it'll run all the tests related to that issue so in Ruby this would be a good
00:28:18.399
time to say for this talk I made four libraries that should never ever ever be
00:28:23.480
really used in production they're just proof of Concepts uh for these ideas
00:28:29.200
that you could just get down and play with um and they're all extensions to mini test because that's what I like to use in Ruby so I made and this is all on
00:28:36.840
my GitHub page so and the links will be at the end so I made mini test Mark for Ruby it looks like this it looks exactly
00:28:43.440
like the python example so the python example turned into Ruby and it does some kind of Unholy things to Ruby to
00:28:48.760
make this even work so you just Mark things and you get a multiple marks and you have the test
00:28:54.960
and then when you run it you just specify it in in environmental variable and it runs that
00:29:00.960
test um for people who want to know this uses uh method added hooks to to do this
00:29:07.720
it's brittle but it works and that's it for marks so the next step uh next thing I
00:29:16.320
want to talk about is paration there's actually a talk uh on test paration and
00:29:21.559
it was good uh yesterday so there's going to be slight overlap um
00:29:28.080
so paralyzation in Ruby uh I never did it when I I did Ruby fulltime and I
00:29:35.559
still do Ruby a lot for vagrant every day and it still seems that paralyzation hasn't really caught on and it's kind of
00:29:42.399
a strange thing because like when I started using python every developer I talked to was like what do you mean they
00:29:47.480
don't you just do it you just turn it on and it just works like you don't have to do anything it just paralyzes and I was like well no it doesn't work cuz like
00:29:54.200
things break and and they didn't understand this concept at all it just like it was news to them so then I was
00:29:59.320
like well why don't we parallelize in Ruby it doesn't make any I mean I know there's a gill but python has a gill too
00:30:05.519
they just make more processes and so I was like why don't we do this so here's a ruby test a syntax
00:30:12.240
error Ruby test um and there's a lot of problems with this test it this was a
00:30:18.159
normal test I think um sh me this a year ago and I been like that's fine but
00:30:23.360
there's actually a lot of problems with this uh the first problem for example is you're you're always opening a file
00:30:30.120
named test file so if you paralyze this test and you have multiple tests then the tear Down's going to destroy that
00:30:35.519
file and some test is going to fail at some point uh a better thing in this case to use would be like a temp
00:30:41.600
file uh another thing is Ruby this isn't Ruby's fault this this is just kind of a
00:30:47.640
standard practice um use a set up and tear down and set up and tear down in my opinion just push shared State and
00:30:58.039
you could use set up and tear down properly but I think it makes it much easier to fall into a trap where your
00:31:03.320
tests aren't easily parallelizable so in this case it's creating assigning state
00:31:09.039
to an instance variable uh you have to create a new instance of the class test class in order to parallelize it um and
00:31:16.200
especially with J Ruby and rubinus it' just be much cooler to instantiate the test once and just tell it to like
00:31:21.919
shotgun out all the tests in that class um so
00:31:28.000
really the general idea the general lesson from that is that shared state is bad this is a general lesson for for
00:31:35.159
concurrency uh paralyzation in general so I think setup and tear down are
00:31:40.360
inherently dangerous um and I don't I don't understand why
00:31:45.919
instance variables ever took off for test State they work but I don't know why that ever became
00:31:51.639
popular so I if you push down all your isolation down into the test level
00:31:57.799
that specific test then it becomes very easy to parallelize and this is what
00:32:02.840
people just do in Python so here's crude attempt number one the first two things I show you are
00:32:09.440
really nasty and you're going to like you'll never want to do this but so let's get rid of the setup and tear down blocks and let's move the state just
00:32:16.679
into each test so in this case it's doing what the setup and tear down block did but moving it into the test and then
00:32:23.080
the tear down as an insure at the end uh of course now you're like well now I
00:32:28.960
have to do that on every single test and setup and tear down was made as a way to to dry up your code and remove all that
00:32:35.320
setup back into these things that are that are hooks so yes I agree this is
00:32:41.440
bad but it's a step in the right direction so crude attempt number
00:32:46.760
two let's take everything all that set up and tear down and let's move it into a method that sets up our state and
00:32:52.919
passes it into us and this just like a file open block this will do all the setup and then do an Ure at the end of
00:32:59.360
the block to make sure that we get proper tear down uh this is not bad um
00:33:06.039
but maybe some of you notice it it just this kind of looks like a function definition right just we're just calling
00:33:13.000
nothing so let's pull that argument up and this
00:33:18.039
better in this case um this is something that uh the python World calls Funk args
00:33:25.120
and what this is doing is passing the state in as a parameter to your test and
00:33:30.440
then it's responsible for cleaning it up after and this pushes all your isolation down into your test uh it's pretty
00:33:38.480
magical I'm not going to not going to try to argue that it's it's pretty magical and I made it work in Ruby um I
00:33:47.360
don't know if this is a good like idea for Ruby but I think Tes isolation is a
00:33:52.799
good idea so there should be a good way to do it in Ruby and the way it works in Ruby is you define some meth method that
00:33:59.799
matches a certain signature so in this case the magic the magic key is to prefix it with mini testor funkar then
00:34:07.279
you do two underscores and then the parameter name and then you return what it's going to what it's going to
00:34:13.200
do then you define your test and when you run
00:34:18.399
it it just works but when you run it what's happening is the test Runner sees that there's a parameter and it's
00:34:23.800
actually named instance and this is where the crazy Magic happens it sees its named instance and then it looks up
00:34:29.599
a test that looks up a method that's available in that scope named that under
00:34:34.919
underscore instance and then it calls it to get the the value um so this is why
00:34:40.040
it's psychotic in Ruby because actually getting the function the argument name is completely non-standard between every
00:34:46.280
uh interpreter every VM in Python it is standard so they do it um and then in
00:34:53.440
python as well I didn't do this in Ruby because I think this example this proof of concept gets the point across fun but
00:34:58.839
they also pass in a method to here where you can set up finalizers uh and all sorts of things so you could do proper
00:35:04.839
cleanup and each state handles its own cleanup and I love Funk args it's very
00:35:13.720
cool and now you parallelize and it's free you just turn it on and it works every
00:35:19.240
time uh and there's a third library that I wrote mini test parallel and you just
00:35:25.000
require it and you run it and it'll actually paralyze all your tests and it's free that one actually if you want
00:35:32.680
to run it in in real it's fine that one's fine um and I don't know there's a
00:35:39.240
lot of tests on a lot of talks on optimizing your tests and parallelization is the easiest way to
00:35:46.520
make your test faster you go from tests taking you know 4 seconds on in a single
00:35:53.000
process to a computer with four cores and they will take one second they they they speed up pretty much linearly in
00:35:59.160
every case and so this is super helpful when it comes to CI just paralyze your
00:36:07.200
tests okay so the third yep the third thing I'm going to
00:36:12.599
talk about or fourth I don't know is performance and when I say performance I'm not talking about test performance I
00:36:19.040
talked about fast versus slow test I'm not talking about test performance I'm talking about performance as a kind of
00:36:24.839
test uh in a lot of applications you would say performance is a feature of the program uh in my work we do
00:36:32.560
mobile stuff so we need low latency so our our limit is an API request cannot
00:36:39.440
take more than 30 milliseconds right now that's our limit so we have tests in in to assert
00:36:46.240
this um so what you really want is an assert it's like assert takes less than forever and then you just run some code
00:36:53.119
uh but unfortunately performance is a different kind of test it's assert are instantaneous uh look at the
00:36:58.760
instantaneous state of something and performance you actually want to look at performance over time it's a it's not an
00:37:06.480
instantaneous view of the world it's a you know time it's a timeline view um so
00:37:13.359
this is the fourth and final Library I wrote called main test speed and it allows you to define a speed test and in
00:37:19.520
a speed test it just runs the test and times how long that test took uh I use a funk Arc here as well and it excludes
00:37:25.920
the setup time the fun card so just the test body and then you run it and just as an
00:37:32.359
example mine outputs the name and the time it took to run it um but what all
00:37:37.720
you really want is something that's easily parsable um for the purpose of graphing it what you want to do is you
00:37:44.560
want to run this on a CI um and you want to graph it and there's a ton of graphing libraries out there and you
00:37:51.480
just pump this data into a database and you graph it and you make sure that it follows a certain curve
00:38:01.280
um so to wrap up integration tests write them first you get a huge value
00:38:07.560
initially um and then the value Trails off and then unit test value starts to rise so but write integration test first
00:38:13.599
then move down to a lower level fast test versus slow test establish a subset of your tests that
00:38:20.720
test every part of your system that are fast enough to run before every commit and do that and move your whole test
00:38:26.280
Suite off to a CI marks they're really cool uh I don't
00:38:33.640
know if there's a good way to do them in Ruby but there's a lot to be learned from
00:38:39.119
them paration test isolation move try to move your your state down into the test
00:38:46.079
level and just flip a switch and start parallelizing you'll biggest biggest
00:38:51.240
speed boost you could get and performance tests performance is
00:38:58.040
a feature of your code at least in some regards so you should start testing that
00:39:03.359
you're getting the right performance out of your code because if your site is taking 500 milliseconds to load then
00:39:11.040
you're going to start losing people are going to start dropping off and that's it so here's all the
00:39:17.760
GitHub URLs to the crazy libraries and really the code in all these is is crazy
00:39:24.200
and um that's it so thank
00:39:55.400
you
