00:00:17.199
how are you guys enjoying the conference so far everyone thumbs up cool all right that's exciting uh so
00:00:24.920
my name is uh Brian Guthrie uh I work for Thought Works uh and uh my partner
00:00:30.679
here can introduce himself yeah I'm amalu um from c42 engineering um we're
00:00:37.879
uh budik rails budik rubby consultancy from uh Bangalore um and I have flown at
00:00:44.039
here from Melbourne and Sushi has thrown flown out here from Bangalore um so I
00:00:49.079
think I I well I don't know I mean I think it's a far away I guess is what I'm saying yeah you have two very jet lag present do yeah I was I was going to
00:00:55.640
try to claim the crown of most traveled but I know you have other C4 c42 guys here from as well uh who have come about
00:01:01.960
the in fact exactly as far as you've come um so oh well anyway so we're here
00:01:07.159
today to talk about Ruby software uh continuously delivered uh and exhaustively explained uh so and I guess
00:01:14.200
by exhaustively explained we mean that we're going to try to cover uh as wide a base of what we think uh you guys need
00:01:20.000
to know for an introduction to continuous delivery practices and Ruby as possible but it's not going to cover
00:01:25.119
everything there are topics we're not going to touch on um there's a great book out there called continuous
00:01:30.280
delivery written by jez humble and Dave Farley which is the inspiration for uh a great deal of this stuff and I recommend
00:01:35.640
that you all run out and go buy it and read it uh because it really is a really good uh read um but this will give you
00:01:41.600
some introduction to a lot of what that book talks about and also uh with a sort of Ruby cast on it um because that's
00:01:48.960
what we're here to talk about and it's what we like doing um so let's let's get going um so by way of uh credentialing
00:01:55.240
ourselves a little bit uh my friend Cy here uh is the author and why I one of the authors and maintainers of a
00:02:01.880
continuous integration tool called Goldberg that c42 has released uh it's a nice simple tool for doing CI um you you
00:02:09.160
guys should all go check it out uh whereas I on the other hand uh yeah Brian H is the primary
00:02:16.239
maintainer currently of uh cruise control RB I'm sure everyone's heard of it but if not it's definitely one of the
00:02:21.560
most popular uh CI servers out there one easily one of the first uh Ruby if not
00:02:27.040
the first Ruby uh CI server it's not a huge amount of development right now but it is actively maintained we both
00:02:32.920
welcome your your commits and your your love and attention um uh they're both
00:02:38.720
nice pieces of software is anyway so we wanted to start out
00:02:44.280
talking about uh where this sort of lies in the spectrum of of release processes
00:02:49.840
um I don't think this should come as as a particular shock to anyone I mean how many people have ever been on a project like this before you have okay so we
00:02:56.800
have we have a few folks right where you have um uh uh an early sort of very staged development sta cycle and then
00:03:04.159
you have a a distinct testing cycle and a distinct uh user acceptance testing so and this thing spans over months and
00:03:10.480
months right this is sort of pre- agile traditional waterfall style very rigid process um
00:03:17.519
and in this model of uh release and deployment you're getting customer value very late in the process right it takes
00:03:24.000
several months it takes you know or even a year um you go through a lot of phases along the way it's very expensive it's
00:03:29.480
very risk prone at one point it was seen to be very rigorous and engineered uh but I think over time as software
00:03:35.319
practitioners most of us have come around to the perspective that uh there's more risk involved rather than um rather than making the process more
00:03:42.280
disciplined it makes it less disciplined uh so we decided to uh move
00:03:48.000
that forward by uh applying agile principles uh so basically we decided to
00:03:54.480
instead of wait waiting for 3 months 6 months to get stuff out there get stuff into production and see the value of it
00:04:00.159
and uh stuff we would do it maybe once every iteration yeah we we integrate
00:04:05.400
constantly it's not like we wait uh we wait near the end of the release to put all the code together no long running
00:04:11.040
branches that sort of thing uh so you know we are we're always uh all the code is always in one code base no branches
00:04:17.600
that sort of thing and uh keep pushing this stuff out to uh some sort of
00:04:22.919
testing environment maybe the testers can pull down the code and test on their own machines uh gather customer feedback
00:04:28.680
uh you know once every few days days uh but and and actually push to production
00:04:33.759
once in iteration maybe maybe a little uh more uh little little rarer than that
00:04:39.000
but something along those lines an iteration being typically one week or two weeks um so yeah and and we thought
00:04:46.720
it was yeah that was definitely an improvement everyone saw value in that uh but as you can tell from our clever use of color here uh and of the color
00:04:53.360
specifically red green uh and yellow uh we're moving in a direction in which all the stuff is continuous right where
00:04:59.400
things just happen all the time where we're trying to get into production as quickly as possible staying within the
00:05:05.240
spirit of delivering value constantly rather than waiting two weeks to deliver that value finding if we can build a
00:05:11.600
process collectively for de for for delivering that value uh in a continuous manner uh and in order to build this
00:05:18.120
kind of process out we really have to be able to answer as a team as practitioners and to our clients and
00:05:24.160
customers I think three questions satisfactorily uh in order to be able to go live all the time we have to know
00:05:29.840
that the code that we're deploying is correct in the sense that the logic that we've written uh actually works the way
00:05:36.280
we intended it to uh and we see that as primarily driven out with a a tdd style
00:05:41.479
process or a bdd style process uh is the deployment process itself safe will going live uh cause any problems for us
00:05:48.440
or will we be able to do it in a stable fashion and is the work that we're actually pushing live valuable are we
00:05:53.680
building out features that uh customers want um are they the right features are we making things visible that shouldn't
00:05:59.840
be visible um pushing the wrong thing out actually has negative value um um and specifically for this uh for
00:06:08.160
this audience uh we're looking at uh what do yeah why is this important to us specifically why is this more valuable
00:06:14.360
to us what do you know what what aspects of Ruby actually uh make this so much
00:06:19.880
more interesting to us you know Ruby has generally been in the Forefront of the Ruby Community I should say is been in the Forefront of uh you know being able
00:06:27.400
to get stuff uh out there as quickly as possible possible being able to uh you know get from idea to a finished product
00:06:35.160
you know that much faster so it there's there's a there's a symbiosis that there
00:06:40.199
there's a real connection between uh continuous delivery and uh and Ruby development but I'm sure there are
00:06:45.440
things we could improve there's there's always room for improvement so yeah we we'll try and focus on that a little bit
00:06:51.199
now all right so uh as part of the way we structured this talk we are going to be playing a couple of personas for you
00:06:57.960
uh and hopefully that will work out okay for us so uhy here is going to play the part of the noble virtuous Paladin style dude um
00:07:06.639
and I am going to be that yeah and it's going to be great um well okay maybe it's going to
00:07:13.400
be it's probably going to be great right all right uh so being the a terrible
00:07:18.680
introduction I'm really got to work on that being The Virtuous uh person trying
00:07:23.800
to be all safe and and be cautious uh you know I would say one of the one of
00:07:29.080
those one one of those aspects that we really you know that's that's really important to uh to continuous delivery
00:07:35.400
is continuous integration you know the thing that preceded uh continuous delivery uh like this is what we talked
00:07:41.919
about earlier about you know ai ai is stuff working out there
00:07:47.120
so I think I've preaching to the choir talking about you know how testing is important but let's Hulk no need tests
00:07:53.759
yeah Hulk code perfect first time all the time Hulk all the time 100% has it
00:07:59.840
never broken in production well 97% perfect first time and what about the other 3% of the time 3% catastrophic
00:08:05.800
failure approximately so what if what if he could do something to avoid that 3%
00:08:11.280
Hulk mistrustful but willing to listen yeah
00:08:17.280
so let's consider th that 3% as uh a half lless Comet out there trying to get
00:08:24.360
through the maze into production so how many of you have seen the show tesh's Castle before or most extreme Challenge
00:08:30.440
so that's the that's the inspiration for the slide um yeah I think so every
00:08:35.880
commit wants to go into production uh you know that's its primary purpose it it's dying to get there and uh it's the
00:08:42.440
only way it feels it can it can fulfill its purpose uh but not every Comet deserves to be in production uh there
00:08:49.360
are some bad comits out there that you don't want to get out there Hulk no like bad commits there we go so so we have to
00:08:56.320
find something to prevent these bad commits from getting into production and those bad comits are sorry and and those
00:09:03.720
gods are tests uh Mr fail here uh hiding
00:09:08.800
uh hiding along the way before the commet can get into production where it doesn't
00:09:14.440
belong uh you like a gold star on production there woohoo
00:09:20.279
sorry um so uh uh someone said it better than
00:09:26.519
we could um so CI is primary CI would every commit in CI would uh be giving
00:09:32.839
rise to a release candidate but the job of the CI system uh is to disprove the production Readiness of a release
00:09:38.959
candidate basically your CI system uh country system should be in place to
00:09:44.560
prevent those uh those bad release candidates from getting into uh uh getting into production and and that
00:09:50.839
should be its primary purpose so that's that's assuming that you aren't already writing perfect code yes but for those of us who are yeah okay we can safely
00:09:57.839
ignore this yeah you don't need it but for the rest of us for the Mortel out here Hulk just making sure
00:10:04.160
yes right uh and okay so let's let's go the different kinds of tests again a lot
00:10:09.440
of this is pretty obvious but let's just make sure we're on the same page unit
00:10:14.800
test uh basically is you test as close uh to the individual modules as possible
00:10:20.760
smaller methods you know just right next to it's we're testing pieces of code
00:10:25.839
pieces of logic in isolation uh and and all by it itself integration test are
00:10:31.079
they working get are they working well with the rest of the code uh and but
00:10:36.120
still you know we're we're talking about the tests are still at the around the same level but it's just a few more
00:10:41.200
layers together uh functional test now we're going a little bit higher uh looking at from the end users point of
00:10:46.440
view stuff still automated but you're going through the browser going you know clicking on things in the web app
00:10:51.880
obviously it be the browser otherwise clicking on Rich client applications whatever it is um and then well there
00:10:57.360
are manual tests we want to avoid the we want to minimize those but sometimes you can't help it there are things that
00:11:03.959
there is value there but you know so have an actual person tested and uh once you actually get the stuff into
00:11:09.760
production you want to make sure nothing is broken so maybe run a couple of reversible kind of tests and if need be
00:11:16.320
have a way of rolling back those those bad comets if somehow they manage to get through all these other four
00:11:22.320
steps um so traditionally this is how uh
00:11:27.360
once again clever use of color y uh so this is how you would this this
00:11:32.600
this is how most systems uh function um very few in unit test if at all um we
00:11:39.720
went back and forth on what we meant by manual unit test I think it's just running the code directly through like IRB or something yeah uh you know and in
00:11:48.560
the narrow sense that you're testing the logic of the code rather than testing it from the user standpoint right so anyone who's ever exercised code through a
00:11:54.200
reppel right through something like IRB to me that's an example of a manual unit test that is something that
00:11:59.920
uh you're using you're purely focused on the code but to to take those and to freeze them turns it into tests that
00:12:05.440
have value um and that well sorry I mean not that rep tests don't have value but that you retain them over time they're stored in Source control they change
00:12:11.560
when the code changes Etc yeah um so uh you know so that's this is traditionally
00:12:17.639
how projects were uh projects were built around but what we want to get to is a
00:12:23.399
lot of unit test uh fewer functional few guy so again this is something I think we all we all already do but the thing
00:12:30.320
to focus on and and this is something I fail to do quite often is when a bug does get into production you know we all
00:12:36.720
we all we know the thing of you know fix write a test for that failing scenario and then actually fix the damn thing uh
00:12:43.519
but I would take that one step further and say if you are if one if your unit test passed but
00:12:50.160
something fails in functional or later um actually try and figure out if there was a unit test missing uh so maybe your
00:12:56.839
uh you know selenium test failed um and there might be a very good reason it cannot be tested at a unit level there
00:13:02.959
is something about the whole you know all all the systems together which means I mean that I mean that you have there's
00:13:08.120
a reason you have functional tests so uh you know maybe this couldn't be tested on a unit level but every time it does
00:13:15.000
fail at a later stage make sure there was uh there was no you didn't miss something and you you know there was a a
00:13:21.760
specific kind of test that you could have written at a higher at an earlier stage in this uh in this build process
00:13:29.800
um so what are we talking about right um when we consider these different stages
00:13:36.680
of the build we're talking about Unit S integration function that sort of thing uh you know if we actually put that into
00:13:43.000
uh an automated process we get a build pipeline which uh again from the continuous delivery book uh Brian talked
00:13:48.680
about earlier uh it's an automated manifestation of your processes uh of your process for getting software from
00:13:54.120
Version Control into the hands of your test right this is stuff we already do I mean so we always had a pipeline I mean we you know mentally we always knew we
00:14:00.839
had to go do this first and then that and then you know maybe it was a read me maybe it's you know stuff that you just
00:14:06.320
happen to know uh Hulk thinks automated manifestation sound very abstract yes
00:14:11.920
looking for more concrete description yes uh so this is how a
00:14:18.560
traditional CI system worked um you know so every comit that went through uh you know went to your Version Control our
00:14:25.240
Cas uh we like GitHub um and your CI keep spoiling for changes you'll not
00:14:31.240
that I cleverly inserted a screenshot of ccrb in there yes say the penalty of not paying too much attention to the slides
00:14:36.800
is that but it's too small I don't think anyone can that's true yeah but I've just called it out so so now everybody
00:14:43.160
knows uh so yeah CS keeps pulling a Version Control sees if there's a new Comet and then runs through those tests
00:14:50.399
that we have uh again before pipelines came around you just had one long one uh
00:14:56.839
one uh build perhaps which ran three rake tasks well I mean to me I think the
00:15:02.199
the the important connection here is that I've seen a lot of CI systems where automated deployment simply wasn't part
00:15:08.440
of the process right the CI was simply a source of information about your tests and the the notion of a pipeline is that you're integrating the next stage right
00:15:15.160
that there is an output from your build that there is a is a deliverable from the result of running those tests which is that you move into the next step yeah
00:15:21.639
exactly um so the next uh so the next stage of this what we we are um
00:15:27.320
introducing here you know from your CI system is to do actually go live with
00:15:32.839
with those changes that you have assuming all the tests pass and uh you know you have complete confidence in
00:15:38.639
your tests and uh you know if you don't go back and write those tests uh but you know assuming you're happy with assuming
00:15:45.519
all the all the test pass of all these different kinds of test pass push stuff to production and then obviously run
00:15:50.839
those smoke tests that we talked about now so Hulk was talking to um to a fellow last night who is uh running
00:15:56.839
through a continuous delivery process and so so we asked him so we assume you have a really like robust crazy test
00:16:02.160
Suite strategy right and the guy was like n um so I I one of the the interesting things that I get out of
00:16:07.600
conversations like that is that you can do this without incredibly robust automated test Suite like we're looking at here um but you are running a risk
00:16:13.519
there um we recommend tests yeah I mean the Temptation is to when you're
00:16:18.920
prototyping things maybe say we don't need tests um and when once it actually
00:16:24.199
you know something sticks and you want to actually proceed with it go back and write those tests you but
00:16:29.519
regard I mean so aside from the fact that tests serve a purpose beyond what we're talking about beyond the
00:16:34.759
regression nature of it to drive design and uh to make sure your code is modular and then all all the rest of it um just
00:16:41.079
from a discipline point of view it helps to uh you know make sure your code is always tested and uh you know so you
00:16:48.000
know everything is uh you stuff doesn't get missed basically all right next yes
00:16:53.839
uh so uh yeah and obviously like we said earlier you would like everything to be
00:16:59.199
automated but there are just some scenarios where you can't uh you honestly cannot test everything in an
00:17:05.079
automated way the Beyond just the fact that you want your uh there might be bugs you want to
00:17:11.880
gather feedback from a the end user point of view maybe the design is slightly off things like that you know you have that middle step where you push
00:17:18.679
stuff to staging I test it production deploy again and and the smoke test right at the end uh the real value of uh
00:17:27.360
the pipeline comes in not just having serial steps where one thing uh flows into the other and me some places you do
00:17:33.039
need them to flow into each other because they're directly dependent on something before uh but uh very valuable
00:17:38.799
thing is to be able to uh parallelize it so that uh you know your your weight time is just as long as your one in the
00:17:45.760
single longest step rather than the sum of all the steps um so uh yeah
00:17:51.440
definitely do that because uh machines are we we say this all the time but we don't necessarily practice it uh but
00:17:58.840
machines are way less valuable or cost way less than people's time so so one of
00:18:04.360
the things I I did want to mention in here is that the inspiration for the term build pipeline is actually CPU pipelining in which you attempt to
00:18:10.679
execute certain instructions um I guess uh optimistically right if you if you're
00:18:15.720
trying to string operations together you want to uh rather than doing them all serially execute them in parallel
00:18:20.840
assuming that you'll you'll find the one that you want uh and and one of the ideas behind build pipelining to
00:18:26.799
accelerate your build is to execute these builds in par assume your unit test will pass and just move on and do the integration the functional at the
00:18:32.200
same time mhm yeah and you know worst case scenario one of them fails it's not like uh you wasted anything anyway so um
00:18:39.280
and a Corel to that I think uh suy I don't know if you want to talk more about this but the the the idea that
00:18:44.480
that the CI server is not just sitting under someone's desk anymore right this is now a first class part of what it is that you do and and how you interact
00:18:50.919
with your software and how your your customer understands what your software is doing yeah exactly I so until now CI
00:18:56.039
has been like a theoretical thing that developers can about it's something we make the only value of it really was
00:19:02.600
that uh when you committed you knew you hadn't broken anything to the extent that other developers wouldn't be
00:19:07.679
blocked from pulling in code and you know working on that but now if you're going to if you're going to rely on it to make sure bad code doesn't get into
00:19:14.679
production it's so much more important and uh you you need that extra level of
00:19:20.840
confidence I mean I think Jim asked this question yesterday about how many people could sleep uh happy tonight if if they
00:19:28.480
knew overnight your code is going to get pushed into production uh you know we'd like to think we would be but how many
00:19:34.400
of us really would and so last night you spent a lot of time worrying about this talk getting pushed into production actually that was was not was not
00:19:40.960
sleeping too heavily in that one uh and yeah and I had a very bad comit just couple of days back where stuff got
00:19:46.840
broken and we had to roll it back so I'm speaking from experience where uh when I'm talking about not having the right
00:19:52.120
number of tests um so yeah basically you have to promote it from something that's
00:19:57.679
just something that developers practice to something that uh your customers are going to care about your end users going
00:20:03.080
to care about you it's it's something it it reaches a whole other level of importance um so yeah you wan to yeah if
00:20:11.480
if uh if you don't mind to ter so if I may move into into um uh shill mode for
00:20:17.000
a moment perhaps that's the right term my employer uh thought Works uh makes a professional CI tool called go um I
00:20:23.559
haven't worked on go myself but it's quite a nice tool I've used it before and the whole point behind it is to attempt to uh take this build pipeline
00:20:30.960
notion and the continuous delivery process and build something that's that's centered around it um so uh yeah
00:20:36.960
I mean if if that's something that you're interested in biom go check it out it's commercial um I think it's free for individuals to use but you get to a
00:20:42.799
certain size you're going have to pay some just thrown it out there yeah I think is for paralyzation multiple agents that sort of
00:20:49.120
thing yeah and uh yeah this is not really doesn't really fit in here but we like uh we like the animation uh so yeah
00:20:56.440
basically your manual test that's definitely the best reason for including a slide in your presentation is if you
00:21:02.360
like the animation yes you type a little bit clicks click this is manual testing by the way should say this an example of
00:21:07.720
manual testing yes uh you type some more click some more you keep doing that and then finally ah eror error uh and that's
00:21:14.559
the QA looks like and uh and then the dev on the other side anyway that's what I usually
00:21:21.159
look like after the QA finds a bug in my code yeah so uh so yeah we just kind of
00:21:27.360
skimmed over deployment uh putting stuff into production let's talk about some more
00:21:33.279
yeah is that what we're proposing yep yes in fact we have a title slide for that um so deployment right if you get
00:21:39.520
nothing else out of this talk um we we do want to touch on the devops thing right uh a lot of where this movement is
00:21:46.880
coming from is from large organizations that have uh more separated uh Devon Ops
00:21:52.880
capabilities um if you're coming from a startup background this probably isn't a problem for you you probably have a bunch of multi multi skilled people who
00:21:58.880
are all physically in the same room and they're all talking with each other constantly at a lot of organizations that's that's not the case um one of the
00:22:06.200
goals of the devops movement and one of the prerequisites I think to doing proper continuous delivery especially at these larger places is to try to bring
00:22:12.919
these folks together uh build Bridges between these organizations and a lot of what I'm going to be talking about in the next few slides is going to be about
00:22:19.440
changing the deployment process and specifically some of the things that Ruby does with deployment to um to
00:22:25.200
attempt to to improve sort of the dialogue that that you have with Ops folks hold on hold on hold on you're
00:22:31.039
saying you don't want people to specialize on on certain skills that seems a little scary yeah you know Jack
00:22:36.320
ofall trades no no no Hulk Hulk life generalists Hulk Hulk smash Specialists I need you to convince me on
00:22:43.159
this go on okay well I don't think you should have a separate deployment team I think if you have a separate deployment team uh you're doing it wrong deployment
00:22:49.840
is something that everyone owns that everyone's responsible for um Hulk smash deployment I think is the the message
00:22:55.159
there but collectively you know we smash it as a as a uh Justice League team as Hulk part of I Hulk not have good
00:23:02.120
understanding of Canon no yeah we have a DC Marvel crossover going on oh okay
00:23:10.240
sorry um but anyway so that's okay though because deployment is super easy in Ruby um and this is another slide
00:23:16.360
that we included just for the animation um so here's how I deploy in Ruby right you know Hulk is all about uh doing it
00:23:21.520
like the the baddest ass way possible so I you know I'm all like I SS route into production right and I'm all like CD VAR
00:23:27.520
site get pull the commit out you know I'm bundle installing I'm rake DB migrating I'm jam it packaging touch
00:23:32.720
temporary start DET text and that's and that's it you know and I just do this on all 18,000 servers um individually but I
00:23:39.279
can do it in parallel because I just get I don't know the Flash in there or something he takes care of it um no no
00:23:44.760
no okay so so right obviously this this process doesn't scale there are good reasons to automate we're all on the same page here right I don't think
00:23:50.080
anyone here is going to argue that automating your deployment is a bad thing except you I don't no it's not in your you're you're pretty cool with
00:23:55.760
automating I think Superman's all about that I just don't want to do it that often it's risky sure yeah well we're
00:24:00.840
all about taking this thing down to that thing down to that thing down to that thing right we think we can do this once every five minutes how does Superman
00:24:07.400
feel about that convince me convince you yeah I I I think it's still risky so Hulk no need convince Hulk Hulk smash
00:24:15.960
Superman no that probably wouldn't work out well for me it's not gonna work um
00:24:21.159
so here's what Capistrano does Capistrano essentially takes something very similar to that that doesn't actually you shouldn't be sshing root
00:24:27.039
into production stop you know doing that but but it takes that whole process and sort of automates it right to the point
00:24:32.360
where you have a script that that runs through all that stuff for you um but to me there's a lot of those things that
00:24:37.640
really don't fall into the category of stuff that you need to be doing as part of a cap deploy there are things in
00:24:43.159
there that um should probably be moved down into your CI survey I think one of
00:24:48.440
the things that we don't do in our CI at least I haven't done on my past Ruby teams is really use the opportunity of
00:24:54.399
having CI to build an artifact and to push that artifact out uh as the unit of deploy and things like resolving
00:25:00.840
dependencies beforehand uh turning your SAS and your coffee script into actual
00:25:06.880
uh you know web languages this stuff can be done and built out in a previous place and doing that right these three
00:25:12.399
things um these are all things that I think can and should happen earlier than that um deployment involves pushing the
00:25:19.520
thing up and then using the new thing uh bundle install doesn't need to be part of deployment converting the things
00:25:25.360
sassin coffee script does not need to be part of deployment um look at build a binary right build a
00:25:30.799
tarball use your operating systems native packaging format um I've even heard of some folks building a gem out
00:25:37.080
of their Deployable artifact and releasing that way you figure rubyists already have a perfectly decent you know uh package management system of their
00:25:42.880
own if you don't want to use your operating systems uh server run your own gem server and then that way it solves a
00:25:48.000
bundler problem too because you always have the versions of the gems that you need available to you that the thing to
00:25:53.039
uh be careful about in this is to make sure your uh CI server CI machine uh
00:25:58.360
already know the staging needs to be as close to your production machine as possible but this will also mean your CI
00:26:03.840
has to be as close to your production machine as possible same OS same Ruby version patch level everything so that
00:26:09.919
uh you know there's none of these problems when you push stuff into right part of the part of reducing the risk of the deployment process if you are going
00:26:15.480
to do this thing where you go out and you build a package is that your CI server should match your prod environment uh as closely as you can um
00:26:21.760
it's it's it's a risk thing I mean I don't has anyone here ever had test failures before that were based on CI
00:26:27.360
environment or testing enironment for PR yeah I'm seeing some some hands and some nods out there um so yeah uh I'm
00:26:33.159
probably preaching to the choir there as well but I think it's an important thing to to remember anyway because I forget it sometimes um and how much do you
00:26:39.480
automate you automate until you require human intervention right until a decision is required so at some point
00:26:45.640
someone should have to physically click the button uh to go out to production uh but up until that point right all all
00:26:52.480
the rest of that stuff can and should be automated um the the actual go live I think can safely be done only by a human
00:26:58.960
who can make the call are we ready yes go ahead and click deploy uh and many of your better release management packages
00:27:04.760
will have some sort of mechanism for that right maybe there's a password involved or authentication or some sort of control around that stuff um a few
00:27:11.960
things that rubyists install in prod right now a few other things that Ruby install on prod right now that probably don't need to be there uh GCC right not
00:27:18.559
for production uh rvm I don't think I love rvm I think it's a great Dev tool I don't think you should be running it in
00:27:24.240
production um get don't need it in production right these are things that part of part of the devops conversation
00:27:29.760
is is uh talking with your friendly neighborhood Ops folk about how to reduce the risk of the things that you
00:27:35.440
push out there and make available live um my friendly neighborhood Ops person within thought works when he heard that
00:27:41.159
I was saying you shouldn't install GCC in production he almost broke down and cried he was so happy to hear me say
00:27:46.240
that um because he and I had thought about that before but I've I've come around I mean I think it's it's one of those things that yeah it seems simple
00:27:52.480
and easy and straightforward but you're reducing the risk profile of your prod servers the less stuff you have installed on them um there's also chef
00:27:59.200
and puppet right so part of uh uh we're not we're not going to get too deeply into this because neither of us are Chef
00:28:04.679
or puppet experts um but um configuration management helps make deployment safe right you should start
00:28:10.440
to think of your servers not as works of art but as things that you're going in there and uh version controlling check
00:28:16.519
everything to Source control make sure all that stuff is managed in a safe way uh and deploy the same thing the same
00:28:21.640
way to every environment so you can build one binary build one tball Whatever deploy the same thing to every server that you're going out to make
00:28:27.360
sure all of those servers are configured the same way using cheffer puppet um this this you can do continuous
00:28:33.559
deployment without this stuff or continuous delivery I should say I suppose um but it's one of those things
00:28:38.799
that helps reduce your risk profile for the deployment stage of this stuff yeah uh how do Superman feel about
00:28:45.919
that I'm coming around yeah think if you automate enough all that risk is gone so yeah maybe we can maybe it's safe enough
00:28:52.519
five minutes five you think five minutes yeah possibly yeah um I did want to call out a tool called the plator which I
00:28:58.720
think is super cool um this is the thing that the ET guys are using to do their uh to as part of their delivery process
00:29:04.120
it's open source I believe it's available up on GitHub um I haven't I haven't used it personally uh but it seems super neat so uh I'm all in favor
00:29:11.600
of things that are super neat um I love the save the princess button there apparently they used to have an environment called staging but then they
00:29:17.960
built out another environment that was also called staging so they had to rename the old environment so they named it princess which as far as server names go
00:29:25.840
uh isn't bad um so the the final thing that we wanted
00:29:31.240
to to touch on and get into and I think that this is probably the most topical
00:29:36.600
topic from a ruby Community standpoint um is is deploying the right stuff and
00:29:42.720
how do we make sure that the thing that we're pushing up into production is valuable how do we make sure that we're not pushing the wrong features out um
00:29:49.679
and so we're going to talk a little bit about feature branches and feature toggles um but before we get into that stuff do you want to talk about
00:29:55.399
mitigation strategies yeah so uh the reason any of this stuff is important
00:30:00.440
either of feature Branch feature toggles is you know you have long running tasks you have stuff that takes more than a
00:30:07.240
few days more than one day to uh get done and you like to be committing all along and that sort of thing maybe even
00:30:13.159
collaborate with others uh and uh yeah and so well do you want to do you want to explain the terms what is a feature
00:30:18.600
Branch yeah so yeah for those who don't know um a feature branch is basically
00:30:24.039
you you create a branch just for one individual feature maybe it's just on your a source Branch yeah Source yes
00:30:29.360
exactly so uh obviously with the git and Mercurial and all these things I don't
00:30:35.240
think anybody uses anything but good um it's it's pretty trivial and yes for you five Mercurial people in I don't know
00:30:41.679
the next city somewhere um I think P Cod con is uh a week down the line so okay
00:30:47.880
well in that City then yeah uh so you know it because branching becomes so is
00:30:54.039
so easy with uh I love Mercurial by the way I don't mean to you know I'm just joking because the sorry because they're
00:30:59.559
not here Hulk totally cool with both Source Control Management Systems Hulk
00:31:05.080
agnostic in that regard uh yeah so uh since uh branching
00:31:10.440
is that easy uh people yeah people end up using it to uh keep your you know you keep the code separate uh the the the
00:31:18.360
stuff that's in process you kind of keep it separate from your master Branch that's going to go into production uh on
00:31:23.399
the other hand feature toggle is you keep the code uh all in all in the one branch and somehow turn off the
00:31:29.320
functionality with a with a with a flag or something which can be turned on if need be whenever it is ready uh so you
00:31:37.320
know and these are for long long running uh long running functional long running development uh stories because you don't
00:31:43.960
want to deploy half big stories uh but you know maybe we don't even need that
00:31:49.039
uh so you know there's a few things you can do one thing is push for smaller features uh you know break every feature
00:31:55.080
I mean you talk to you talk to the people defining these features to see if you can break it down into a smaller
00:32:00.240
task that can be done uh uh and you know you get value as you're building it rather than wait for a for a big thing
00:32:05.720
is almost always a way to uh do that I mean even if you have big features you
00:32:10.760
know make it make them into smaller tasks Each of which like I said gives you value uh Each of which is testable
00:32:17.080
and you know on its own and it's not going to break anything um and uh yeah it's it's a little hard to do the bdd
00:32:23.200
without you know little hard to do with bdd because you know it bus actually you know it's
00:32:28.639
business driven that but you know you want so so what do you mean by business driven then why is it hard to do with
00:32:34.399
PDD because it's business driven because until you have the UI I mean so basically what we're saying is to add the UI last so that you can do all the
00:32:40.399
other stuff and nothing is reflected on on the screen but if you if you do that
00:32:45.639
uh so the bdd approach is is top down right you start with the UI you drive your way in ex exactly and and and and but if the business doesn't actually get
00:32:51.880
to see it uh you know it's not really business-driven anymore so the your you know your uh mileage may vary on that uh
00:33:00.159
and uh finally uh yeah Branch by abstraction so Branch by abstraction is is the notion that you can insert an
00:33:06.320
abstraction layer between uh the pieces of your application
00:33:11.880
um gosh how do explain that we should have had a diagram for that because that requires a diagram but the notion is
00:33:17.320
that if you have a really big change that's coming into your application say you're changing the orm layer you build an abstraction layer between your
00:33:23.080
application and the orm layer um and allow that abstraction layer to toggle between the two inter alternative
00:33:29.320
implementations build out one while you continue to maintain the other and then eventually you're in a position where you can swap over from one to the other
00:33:36.000
I think of feature toggles as being kind of a special case of that but but Branch by abstraction is really one of those
00:33:42.000
Concepts that's very helpful for large difficult changes um that having been
00:33:47.200
said I don't think you could do Branch by abstraction with something like a rails upgrade um I've done uh nasty
00:33:53.080
rails upgrades before and that's you can't abstract away the framework right yeah exactly um anyway yeah so uh the
00:34:00.159
gith up flow model which I as far as I know they use for everything yeah so I hope I hope this is correct cuz we Tracked Down Zack hulleman last night at
00:34:07.399
the um at the the amp block party and physically assaulted him with questions about this uh to make sure it was okay
00:34:13.839
but we didn't actually show him the slide but this this reflects sort of our understanding of how they're going about building features into GitHub and also
00:34:21.200
should be pointed out that we were all possibly a little drunk so yeah that's okay so we'd had a couple beers by that
00:34:26.560
point and then we had to identify who who exactly you know he was by his Twitter photo which was fun too going
00:34:31.839
around I have you seen this man you know showing the iPhone um but eventually we did in fact find him and he was very
00:34:38.240
gracious so thank you Zach yeah uh he's not here I don't think U yeah uh but uh
00:34:45.679
yeah basically the way it works is you know you have your master you Fork from uh you know Fork each of your branches
00:34:51.560
from there um and uh you know work independently of everybody else once it's actually done uh you know a day
00:34:59.240
down the line couple of days down the line or even maybe within the same day uh you give a pull request or you maybe
00:35:05.000
you push you're assuming you have access and that's the process you push directly into uh into master um and uh yeah and
00:35:12.680
then it obviously go goes into production uh the way this works with multiple people is you are um you know
00:35:19.599
so say Brian starts on his on his uh search form functionality uh a little
00:35:24.880
bit down the line I decide to work on a on a separate uh piece of functionality which is a different search form uh you
00:35:31.320
know I maybe I start a little bit later but I don't have his changes halfway through my uh halfway through my working
00:35:36.640
on it I still don't have his changes although he's been making comments on his Branch I I don't understand that
00:35:41.760
release bar there though I don't I feel like that Arrow yeah so but I'm I'm rebasing from
00:35:47.560
master I have all these other changes oh rebase that's what it says okay yeah um and I'm getting uh changes from uh
00:35:53.680
Master throughout but you know I'm just not getting any any any work done any of the work done by Brian on his
00:36:00.280
branch and any refactoring I'm doing along the way is going to get you know broken by or is going to break when
00:36:07.160
eventually I have to merge back merge back into Master once my features actually done so this is a general issue
00:36:13.200
with any any feature branching strategy that relies on relatively long running branches my understanding is that GitHub
00:36:19.400
manages to avoid most of this stuff because they IND they they run CI independently in each branch and they tend to not leave them around for very
00:36:25.400
long they're not long running branches they get them in quickly but uh this is there are certain kinds of semantic
00:36:31.359
merge conflicts that git as awesome as git is uh is not capable of resolving um
00:36:36.640
in practice they don't happen all that often I don't think but when they do happen they tend to get pretty nasty and that's when two people independently
00:36:42.240
refactor the same piece of code or change it in some other meaningful way and are later unable to resolve those
00:36:47.440
differences without a lot of uh painful munging and crushing and uh General
00:36:52.720
Agony um in the case of defu renamed to either bar or baz that's not a whole lot Agony as far as these things go that's a
00:36:59.160
pretty mild like sort of mosquito bite level thing unless you have 18,000 gazillion people calling that method
00:37:05.359
exct in which case you're in trouble yeah but yeah you try and rename model or something and have everybody reing to
00:37:11.599
the old place it's just things just go out of wh if if you start doing migrations in uh individual branches
00:37:18.640
that's you know and if if they're somehow affecting each other then you just kind it's not it's not a Version
00:37:23.839
Control conflict now but it's it's something much more fundamental you actually might have to go back can rewrite those migrations and somehow
00:37:29.520
make it work so but but as Zach and I think the rest of the GitHub guys would point out uh communication is a good mitigation strategy for this right if
00:37:35.240
you're talking constantly if you're chatting if you're making sure that if you're undergoing larger refactorings that you're in communication with the
00:37:40.280
folks around you you can prevent folks from getting off hopefully too far into the woods with this stuff yeah so but
00:37:47.119
you know to kind of avoid this and and maybe not do this all the time uh we
00:37:52.839
could go with the other way of uh the other way the other way so oh yes
00:37:57.880
there's probably click that slide there um so most of the folks that I work with are are big fans of the strategy which
00:38:03.760
is a very traditional CI oriented just commit everything to a trunk or Master as the case may be
00:38:10.160
um the spirit of CI is that everyone's changes are visible to everyone else all
00:38:15.400
the time um that everyone has the latest version of the code that no one's out there working in isolation for days and
00:38:20.480
days that you're constantly merging into the same spot um and the folks that I've talked to seem to think that this is
00:38:26.160
especially as your team starts to grow out a safer strategy for doing uh continuous delivery that um the more
00:38:33.440
people you add to the team the more risk you're running that someone's going to uh run into sort of long running uh
00:38:39.359
feature branches that go a little longer than they should the harder it is to coordinate and communicate around what changes might be occurring um you know
00:38:46.520
this is not without its problems right but in this particular case the diagram is trying to show that well okay so B Guth a commit he pushes in a master Cy
00:38:52.960
rebases he gets that commit brings it back down again and the other nice thing about using this rebase models opposed to emerge model is that you end up with
00:38:59.119
very straightforward um commit history right it's very very simple there's not a lot of arrows heading in and out of
00:39:05.200
master um I happen to really like this I know gits really powerful in that regard I don't think that that's the kind of power that is necessarily helping you
00:39:11.800
much um but again it's not without problems right so if I insert a feature toggle and other people insert a feature
00:39:17.720
toggle eventually you're going to get you know merge conflicts right um I I think that's what that's trying to say
00:39:23.040
there um the the for for long running features for things that you do not want
00:39:28.280
visible to the user interface that's an example of what I'm uh what I'm talking about right down there right is if if if
00:39:34.440
the Masala feature is enabled then we render a masala dosa uh which is more
00:39:39.480
delicious than a plain Dosa I happen to think um got some agreement from the audience there um so feature toggles the
00:39:48.520
trouble with feature toggles and the and the trade-off that you're making there is that they tend to get crufty right they accumulate you have to be very aggressive about removing them um if
00:39:55.000
they if you don't remove them promptly they can accumulate in nasty ways but
00:40:00.440
the big you get two big benefits out of them though right and one big benefit is that you start to have a conversation about uh enabling and disabling certain
00:40:07.640
features um feature toggles aren't quite the same thing as enabling and disabling features I mean that's sort of a
00:40:12.720
separate category of of you know if you want to be able to turn off email in your in your system or whatever that's sort of a that that should be
00:40:18.880
intentionally built in and there should be a mechanism for that but you can start to have a conversation with the business about what what parts of our site can degrade gracefully and how can
00:40:25.880
we build out a process for doing that uh and the other huge advantage of of um
00:40:31.280
uh Feature Feature toggles I think is that you start to decouple deployment from release so the deployment or
00:40:37.680
release are no longer the same activity uh you can be deploying code and features that while it's physically
00:40:44.640
present in the codebase uh aren't yet released out to the public um and that is another strategy for making your
00:40:50.079
deployment process safer I think in the long run um I the the thing to be careful about is there's some so I we
00:40:58.040
showing a very very trivial uh example of a feature toggle here something in The View with either Enders or it
00:41:03.400
doesn't but it could get a lot more complicated than that and you might even miss some aspect of like there's some
00:41:10.880
calculation which now has to consider a new kind of tax or a new kind of some new uh number there and uh code just
00:41:18.280
starts to get really complicated your tests around it because you have you'll be you'll have a test for the for the
00:41:23.599
situation without the uh without the where it's toggled off another one where it's toled on and it's uh yeah and so
00:41:31.400
like there's the limitation obviously of screwing it up where um you forget to toggle something off further down the
00:41:36.960
line which is not immediately apparent on the screen and now you have something half finished into in production uh but
00:41:42.480
yeah your test start to get really complicated as well so couple of negatives there yeah uh oh and and well
00:41:50.040
did you did you address testing positive and negatives of feature toggles as well yeah you should write tests to ensure
00:41:55.160
that you're testing with the application with them enabled and disabled uh in both States otherwise you're not fully covering your yourself I think yeah and
00:42:02.200
and yeah if it just makes it easier for if uh your staging environment if you can turn it on and off you you can you
00:42:09.160
have feedback on features as it's being developed unless you set up another uh
00:42:14.319
staging environment for every single uh feature branch that you uh build which you could but just adds that that much
00:42:20.720
more of a uh an Ops overhead to everybody uh but if you you know if it's
00:42:26.920
if it can be turned and off trivially you can get gather feedback even as even if before the feature is finished and
00:42:32.760
you can go back and fix it and you know as we all agree I think quick feedback is always
00:42:38.520
better I was going to make a Hulk joke about that but I think Hulk prefers quick feedback as well um so finally I
00:42:45.240
think the final piece we wanted to talk a little bit about here is uh in the in the git Community there's developed a
00:42:50.720
bit of a um a taste for really Baroque uh branching and merging models
00:42:57.720
um I think Baro is a good way to put that did I this this seems to be working
00:43:03.000
for does anyone follow a release process like this like the gitf flow stuff how do you how do you guys like it not with
00:43:10.640
perforce okay apparently the the fellow using
00:43:17.599
perforce is not from Seattle we've confirmed um this this appears you like
00:43:24.839
okay I mean it appears to be working well for a lot of people it works when you have releases of longer
00:43:31.480
running okay and oh yeah that so so so with long running releases you have to
00:43:36.599
be able to maintain the Past release and apply bug fixes to it and so forth I I
00:43:41.920
think that as you move to a more rapid uh delivery model the the usefulness of
00:43:47.640
maintaining those separate branches tends to go down quite a bit um and it becomes more complicated than you really
00:43:54.240
would like um so it's it is a really handy release process and I think that's
00:43:59.720
that's a good point to call out that it's it's definitely it's it's hard when you have a longer release cycle it's
00:44:05.520
hard not to end up with something that looks a little bit like this I think as you try to maintain older releases uh but as you get more and more into the
00:44:12.319
Rapid Release cycle stuff uh simple um releases off of Master are probably the
00:44:18.160
way to go um yeah um and then we had a discussion earlier about what exactly
00:44:24.079
we're trying to say with this slide um and I think I was trying to make a pun about live people and dead people in
00:44:30.000
soent green um unfortunately soent green is people but so green is people but they're dead people so sorry if you gave
00:44:36.280
a spoiler alert yes turns out long story um uh so
00:44:42.839
anyway I I think the message there soent green aside is that um to do this
00:44:48.119
properly you need to be in communication with people you need to be talking with the folks in your team you need to be reaching out um it's it's not just a
00:44:54.800
technical topic right we've we've talked a fair amount about the technical stuff uh but the hard part about this is the people part as always with software uh
00:45:02.319
and I think we have a couple minutes for questions yeah not too much time uh the question was how does code review fit
00:45:08.680
into the everything on Master model uh most of the teams I've been on that committed everything on Master have done
00:45:14.319
so using paer programming and pair programming you have instant peer review one of the big benefits of the GitHub
00:45:20.599
full model and the notion of using pull requests is that the pull request acts as a peer review mechanism right other
00:45:26.640
people review the stuff that's coming into your code base before it actually gets merged
00:45:32.920
um to me pairing is a huge benefit it's something that you should consider doing
00:45:38.280
if you're not not Everyone likes pairing not everyone gets into it but to me I think it's a really useful feedback
00:45:43.480
mechanism um I might also lean towards uh so code
00:45:49.920
review before merging into Master might imply a certain lack of trust in the code uh so I would say push Master maybe
00:45:57.640
review it after it's done but you know like you don't have it doesn't have to happen before the merge and you can always roll it back if there's something
00:46:03.839
wrong but you know more often than not your tests and you just basically having good developers should avoid a lot of
00:46:10.760
these problems there's anything gling you know fix it later rather than Step One is have good developers yeah so that
00:46:17.200
and that's the trust but verify model right that that allow people to commit but but do a little bit of verification
00:46:22.240
afterwards yeah um back there uh okay okay so that
00:46:27.280
was that was a double question right there um the first was uh have you guys found any gems to help with uh the
00:46:33.440
feature flipping stuff and the second was why did the Hulk get strangely articulate all of the sudden um and I
00:46:39.480
think the answer to the second question was sheer convenience um because I think it would have gotten really annoying to
00:46:45.680
speak like that for really everyone involved past a certain point um as far
00:46:50.880
as the is that does that square with your yeah oh you could just say you went back and forth with Bruce B in the oh
00:46:57.559
that's true yeah Hulk Hulk calmed down transformed back sort of chilled out a little bit you know talked through the
00:47:03.240
pros and the cons um as far as gems go there are some good gems out there for feature flipping none of them are really
00:47:09.160
super complicated or involved that I've seen um they tend to be fairly straightforward you have a configuration
00:47:14.559
file yaml file of some sort and uh uh some sort of global mechanism for asking
00:47:20.280
whether or not a particular features enabled or disabled I can't remember any names off the top of my head but I know they're out there because I've I've done
00:47:26.480
a little resarch into that before do do you know of any roll it's really simple though it's
00:47:35.119
done my company is done a bunch of future flipping and tried to use other libraries but it's one of those things
00:47:40.839
where the implementation is so straightforward seems to be so tied to your particular application that really
00:47:48.359
at a certain point like
00:47:56.960
extremely fast to write to and from and we have all kinds of different roll outs roll out as a new user signs up roll out
00:48:04.280
you know all right I think we we might be
00:48:10.440
out of time I see people wandering around and moving but thank you all thank you all very much for coming thank
