00:00:11.120
thank you everyone for coming to my talk Um this is a big honor to present at
00:00:16.320
Ruby Kai That's my second time in Japan and second time at Kaii and I'm very excited to be here We'll be talking
00:00:23.760
about Basil for Ruby and we'll really talk about three things What is Basil
00:00:29.599
why what is actually is uh how do you use it in Ruby projects and what's the
00:00:35.840
current state of the Ruby language support in Basil ecosystem Uh my name is Alex You can
00:00:42.079
find me as PJ on Twitter GitHub LinkedIn anywhere I am uh a Ruby maintainer at
00:00:48.879
the Selenium project a browser automation tool and I'm also a member of the Basil rule authors special interest
00:00:57.440
uh group and I maintain the Ruby language support within the Basil
00:01:03.080
ecosystem So what is Basil and you might be familiar with tools like make rag uh
00:01:10.880
but Basil is essentially like that It's a build and test tool and it it's a tool
00:01:16.640
that you use to build software It's a tool that you use to run tests on your software It was created specifically for
00:01:24.680
monorapus and it was created at Google uh under the name blaze and then open
00:01:30.240
sourced under basil and uh Google is known for having
00:01:35.759
this mega massive monor repository uh where they host all their different
00:01:41.759
software packages which interact with each other Um because of that because
00:01:48.159
basil is created from mon repositories It was designed to be polyglot
00:01:53.200
supporting different programming languages Java Python are working out of the box but through the system of
00:02:00.159
extension modules you can add other languages support like go javascript and
00:02:06.079
pretty much anything uh being a project created at Google where you have thousands of engineers working with the
00:02:13.360
codebase you can imagine basil being very efficient and very scalable and um it was created in in in
00:02:21.599
a certain manner so that it's heratic meaning that it will actually install
00:02:27.120
all the software that you need in order to build your so to build your programs
00:02:33.200
and reproducible So regardless of what machine you write you run Basil it's
00:02:39.440
supposed to produce the same result by making sure that it downloads the software that is used during the
00:02:45.879
build And um if you want to create a build tool that's used by thousands of
00:02:51.760
engineers you have to build it with distribution in mind So things like uh
00:02:56.879
build caches and the actual execution of the build processes is uh happening
00:03:02.800
locally on your computer but it can also be uh done remotely in a distributed
00:03:08.640
systems and that's pretty much how the builds are working in in all the build in all the large uh companies using
00:03:16.599
basil Okay But uh if you if I want you to take something out from this
00:03:21.640
presentation just keep in mind that basil is a building and test tool and that's pretty much it So what is
00:03:27.360
actually build So what happens when you run basil build um and let's say you have a Ruby source Ruby project where
00:03:34.080
you have some uh source code files like remove add or B and then you want to package it inside your gem and you have
00:03:41.280
some gem script that actually you need to run So you run gem build and then it
00:03:47.200
creates a new file called calendar.je So you can split this whole
00:03:52.239
process into three phases You you have some inputs which are source code files It could be environment variables and
00:03:58.720
maybe other things But you have inputs You run some action like a command that
00:04:05.519
runs over this inputs and that creates a new output And of course you can use the
00:04:10.720
output of your built target to be used as inputs to other build targets So
00:04:16.639
let's say you create this calendar.je and now you want to package it inside a docker image So maybe you
00:04:23.680
use it as an input for a new action uh along with a docker binary and u a
00:04:30.560
docker file and that creates a new output uh a docker image which is a uh
00:04:36.560
an archive and uh that's how you create the build commands you take inputs you run
00:04:43.840
some action create new outputs that's all what happens within the build and I
00:04:50.160
said that specifically that basil is build and test tool and test in that
00:04:55.199
regards is not completely different So you also have some source code files uh
00:05:02.160
maybe your Ruby classes and maybe some specs that you use uh to test this Ruby
00:05:08.320
classes and the RSpec script And then these are all inputs and then you run
00:05:14.080
RSpec an action over these inputs and that creates uh an output result which
00:05:19.520
could be a test result with some logs and the exit code and exceptions if if
00:05:25.440
they were thrown And I hope you start seeing the pattern but it's
00:05:30.919
essentially all the source code that you are dealing with and all the different targets you have to build and test are
00:05:39.759
it's essentially a graph It's a to be more precise directed cycling graph
00:05:45.280
where your lift nodes your inputs are the source code files and then you have a different
00:05:51.919
commands and different outputs and that's pretty much what Basil does
00:05:57.280
internally when you work with it So it builds this massive uh graph build graph
00:06:03.600
of uh your source code and the different targets and then it allows it keeps it
00:06:10.639
in memory and then because it keeps it as a graph it can do pretty awesome things uh by looking at this graph So
00:06:18.639
for example it tracks uh all the different inputs uh and see and knows
00:06:25.199
when they're being changed So for example if you change the source code in
00:06:30.240
the spec file Basil knows that it needs to run specs because our spec uh actions
00:06:39.039
are taking this spec file as an input But it also knows that it doesn't need to do gem build or docker build because
00:06:47.199
uh the spec file is not an input to any of these targets And because it keeps
00:06:53.120
all this in memory and tracks all that it is very efficient and it really knows
00:06:58.960
what needs to be done and what else does not need to be done And it is hermetic So in order to
00:07:07.039
to be really efficient Basil also needs to keep a track of all the different
00:07:12.639
software and scripts and binaries that you use during your execution So things
00:07:18.240
like the and I I kind of oversimplified here saying that bin docker or bin gem
00:07:23.440
is all it needs but it keeps track of all the different um uh binaries that
00:07:30.479
you need and then when for example you update the version of Ruby uh or maybe
00:07:36.560
the version of Ruby gems it knows that bin gem has changed So now I need to
00:07:42.639
rerun the gem build uh because the inputs have changed and you can see
00:07:50.199
how it's really doesn't matter what kind of language you use it's polyglot and
00:07:56.240
maybe if you have something like a JavaScript minification pipeline uh some
00:08:02.160
JS files and then you want to run some ESB build commands over it that might create a new output with a minified
00:08:09.440
version of JavaScript source code and then put it as part of your docker image
00:08:15.120
again it's part of the build graph from basil perspective it's not different in
00:08:21.840
any way from any other targets When I say that Basil is polyglot it
00:08:28.720
truly is Um pretty much any programming language
00:08:34.640
is uh either supported out of the box in Basil or is available through the module
00:08:41.399
system like Java Python NodeJS and different dialects like TypeScript
00:08:48.080
uh compiled language like Go Rust uh and even specific tools that you use during
00:08:54.240
the build process maybe the doc docker or Helm or Terraform or something And
00:08:59.360
these are all uh supported in Basil and there are usually some kind of community
00:09:04.880
support uh community extensions and we're here at Rubiki So
00:09:11.360
we'll be talking about rules Ruby Rules Ruby is a canonical rule set within the
00:09:16.880
basil ecosystem that adds support for Ruby programming languages So you can use rules Ruby to build uh uh to build
00:09:25.279
Ruby software to run the tests for it uh it was originally created as part of
00:09:31.839
my work in the selenium project but it's now maintained by the basil rule authors
00:09:37.519
special interest group so it's part of the broader u basil
00:09:43.000
ecosystem and uh it's available in the central registry so basil has a
00:09:49.120
something like ruby jams where you have all the officially uh endorsed canonical
00:09:55.680
modules are being hosted and where different versions are hosted and Basil will automatically install it from this
00:10:02.839
registry and it's currently being used in production in many different companies Most of them didn't want me to
00:10:10.959
put their logo here So thank you Stripe for letting me do that Um and a lot of
00:10:16.720
open source projects also use rules Ruby internally like uh Protobuff from uh
00:10:22.399
Google and Selenium and I have a small QR code here uh if you want to check out
00:10:27.519
the GitHub repository So what does rules Ruby give you when you start using it with Basil
00:10:35.040
First of all it allows you to install the Ruby interpreter Um and out of the
00:10:41.040
box uh we it supports C Ruby J Ruby Truffle Ruby and in fact pretty much
00:10:47.279
anything uh any other interpreter uh it will install it with basil and put it in
00:10:53.040
a special location so that basil keep track whenever uh something is changed within the interpreter version It allows
00:11:00.240
you to install dependencies with bundler Again basil needs to keep track of all the different gems you use in gem file
00:11:07.440
So it will download put it into a special place and uh watch and then uh
00:11:14.640
in order to actually use basil you need to create the targets the build and test targets and uh rules Ruby provide a
00:11:22.240
certain common uh operations like uh our build Ruby library which is a way to
00:11:29.279
expose the source code of Ruby files into the build graph uh binary which
00:11:35.279
allows to run Ruby scripts It's test which allows you to run the test commands like our spacest
00:11:42.839
rubocop and something specific to Ruby ecosystem like building gems pushing
00:11:48.079
gems There are a few other things uh but these are the most uh important
00:11:53.800
ones Okay so you there is rules Ruby but how do you actually use it in Ruby
00:12:00.120
projects And let's take a look at the simple gem uh like a calendar gem where
00:12:08.399
you have a fair fairly familiar structure You have a gemspec gem file with dependencies Um all the source code
00:12:16.079
lives in lib folder and there are some tests in spec And if you want to start
00:12:22.680
building this jam and testing it with Basil the very first thing you have to
00:12:28.880
do is add a module.basil file It's uh you put it into the root and it's kind
00:12:34.720
of marking the root repository um of your of your package And uh from
00:12:42.160
here uh you use a special language It's called Starlark which looks a bit like
00:12:47.519
Python but it's not really Python and you use it to say that you need rules
00:12:54.320
Ruby installed and then you want to load this uh rules Ruby as an
00:12:59.480
extension You use it to install uh the interpreter and here we say that give us
00:13:06.320
version 3.4.2 Um but you can use you can use it
00:13:11.680
also for J Ruby or Truffle Ruby Um you also tell that you're going to be
00:13:18.000
downloading some dependencies and installing them with bundler and we
00:13:23.200
refer to the gem file and gem file log for that And finally just tell basil
00:13:28.560
that we're actually going to use it all of that throughout the build So
00:13:34.839
um that will install the uh interpreter install the dependencies but how do I
00:13:41.120
actually build it How do you actually use Basil to build things And in order to do that surprise you need to create a
00:13:48.639
file called build which is a great name Um and again we're here using this
00:13:54.399
Starlark Crow language But the first thing that we want to do let's say we we
00:13:59.839
want to create a gem from from all this source code Uh we load rules Ruby and we
00:14:05.600
say give us the special function for building the gem And then we define a
00:14:11.040
build target And every build target has a couple of uh common attributes The
00:14:17.120
first one is always name This is the name we were going to use in our uh command line
00:14:23.079
invocations And there is this special argument called deps which uh tells that
00:14:28.720
this particular target depends on uh the targets inside the lip folder You can
00:14:35.440
see there is double slash uh in front of it which in basil is just the way to say
00:14:41.199
start from the root start from where you have module.basil file and we say that
00:14:47.279
we're going to build a gem called calendar We're going to use a calendar.jpspec as a gem specification
00:14:53.920
file and this build target is going to depend on everything inside lib So it's
00:15:00.880
going to be part of the gem build process And if we try to
00:15:06.079
run it you run it from command line like make rake You say basil build double
00:15:11.920
slash it start from root and then you say uh colon it's a way to to to
00:15:19.000
to delimmit uh everything after colon is a is a name the same name that we used
00:15:25.519
in the name attribute So we say build from the root target name calendar If we
00:15:32.320
try to do that we're going to get an error because uh we depend on lib on all
00:15:38.560
the source code in lib but we don't actually have a build file there We have to add it in order for basil to know
00:15:45.360
that it's part of the build graph So let's fix that Let's add a build file inside lib folder And again we are here
00:15:53.519
loading rules Ruby and we load this Ruby library function which is a way to
00:15:58.800
expose Ruby files to the build graph And we say that our library is going to be
00:16:06.160
called name And it's going to be using just one file called calendar.rb It's going to be publicly
00:16:13.199
visible because basically you can actually mark certain parts of your code
00:16:18.480
and files private Um and then it's going to be depending
00:16:24.160
on the source code inside the calendar subfolder So things like add and remove
00:16:31.199
And if we would try to build the gem at this point again we would get the same folder because now we need a build file
00:16:38.079
inside calendar subfolder So let's cut it short and let's add uh a couple of
00:16:44.560
targets here uh a one that will just expose the add rb and the other that is
00:16:50.880
going to expose remove.rb They're both going to be publicly available and uh no
00:16:57.800
dependencies for these particular files So now when we run basil build calendar
00:17:03.600
it will actually take all these dependencies take all these files put it in a special
00:17:09.760
folder uh kind of isolate it from all your operating system software and all
00:17:15.600
other files that you might have in your repository and then run gem build over
00:17:20.640
all of that And that will create our gem file and put it into a special folder
00:17:26.000
called basil bin which you can then use to maybe unpack the gem or do something
00:17:31.919
else that uh it's just a gem file Okay so building the gem is at this
00:17:41.520
point done How do we for example push it Uh let's go back to our build file in
00:17:46.799
the root folder and we're going to add a couple new targets We're going to load a function called Ruby gem
00:17:53.320
push and we're going to create a new target called calendar release And this
00:17:58.559
target has just one attribute called gem And uh here we we use the same uh syntax
00:18:07.280
colon calendar In this way we say that in order to push the gem we're actually going to be use the gem that was built
00:18:14.400
before that was built in the calendar target And that's uh and that would internally
00:18:21.360
in basil make sure that the build graph means that calendar release depends on
00:18:26.400
calendar and kind of the gem file that it creates as an output So now we can
00:18:33.039
run this command because pushing gems is does not produce any artifacts It's not
00:18:38.559
really basil build It's basil run command You do it It has some side effects like push into network and
00:18:45.840
building by default uh is completely sandbox You don't even have network access So we run the calendar release
00:18:53.360
and Basil knows that in order to push the gem first we have to build the gem
00:18:58.559
So it does that and then it would invoke the gem push and uh and make the the gem
00:19:06.160
publicly available So uh that's kind of how you work with
00:19:12.000
building Ruby uh with Basil You just start from the root and you start adding
00:19:18.160
the build files and kind of building the dependency build graph and at a certain
00:19:23.440
point you have uh a complete coverage across your mon
00:19:28.840
repository So this covers the build What about test And in order to run tests
00:19:35.840
with Basil again we add a build file So let's go to the spec and create a build
00:19:41.440
file there We are loading the Ruby test function which uh is a special way to
00:19:47.679
kind of say these are test targets And we add a couple of those One
00:19:53.120
for add spec RB and we say that it's going to be using the RSpec script And
00:20:01.039
in the dependencies attribute we say that in order for this test to be run we
00:20:06.160
actually only care about the source target add We don't really need anything
00:20:12.960
else in our source code in order to run this particular test And likewise for
00:20:18.480
remove uh we have a similar approach just have the remove spec remove.rb B
00:20:25.120
and there is a special syntax because we use RSpec We actually need to load the whole um gem file
00:20:32.919
bundled dependency uh dependency graph Um and then we invoke it through another
00:20:40.960
command called basil test Again starting with double slash from the root and we
00:20:46.880
use a syntax with triple dots with in basil is a way to go run all the targets
00:20:53.760
uh recursively from this root So test run all the test targets that are currently present in this
00:21:00.200
repository And what's going to happen internally at this point is Basil will
00:21:05.440
create two separate sandboxes uh take the files that are part part of
00:21:10.640
the dependency for add take uh the files from remove uh put it into two separate
00:21:16.880
sandboxes and run our spec in parallel to each other in in all of these sand
00:21:21.919
boxes As long as everything is great it's going to say that a couple of tests
00:21:27.440
were run They're passing in in in in a second And that's how we run this kind
00:21:34.960
of tests Um the last uh thing I would like to mention is how do we for example run
00:21:42.240
rubocop and we could follow the same approach creating a separate uh test
00:21:47.760
targets to rubocop specific files But for the purpose of simplicity let's say
00:21:52.880
we just want to have one way one target to run rubocop So we go back to our
00:21:58.240
build file in the root We add the new uh Ruby test uh target Let's call it
00:22:05.240
rubocop It's going to use gem file because rubocop can l gem file It's
00:22:10.799
going to run the rubocop script And then in dependencies it's going to be using all
00:22:16.400
the source code from lib and all the source code from specs and of course the
00:22:22.159
bundle because we need we needed to run the rubocop So when we run this test target
00:22:29.520
basil will create a sandbox put all the dependency source code files there and run rubocop
00:22:36.080
And if we run the test again with the same arguments uh you can see that in the bottom we have a new result from the
00:22:42.880
rubocop which is again passing But what one thing that I would like to highlight
00:22:48.240
here that you might notice is there is this cached word So what does it mean
00:22:55.039
Basil knows that we have created a new rubocop target but we didn't really
00:23:00.080
change anything in the source code and it already run the tests before So it
00:23:05.760
knows that it doesn't really need to run any kind of tests any specs So it uh
00:23:11.760
just loads the results from cache and says hints you it's it's a cached result
00:23:16.799
You can still inspect it see the logs and everything But these tests were not really run They they're instantly loaded
00:23:24.240
from cache And this is a very powerful thing
00:23:29.280
which I would like uh you to keep in mind when working with Basil as long as
00:23:34.320
you create the proper build and test graph and you have test targets that
00:23:39.760
depend on particular uh that really only uses the Ruby code that they're testing
00:23:46.000
You can have thousands of tests running in seconds because most of the time you
00:23:51.919
don't really need to run all the tests Basil will only run uh things uh that
00:23:58.080
are necessary to be run based on your changes So the current state of rules Ruby
00:24:06.799
um is that it's it's it's an early project um but a lot of great things
00:24:14.000
that Basil gives you out of the box uh are already there which you can use So
00:24:21.200
the specs rubocop these are all being run in parallel isolated from each other
00:24:26.880
So if you use rules ruby and you start testing the projects with basil you get out of the box an experience similar to
00:24:33.600
something like parallel test gems because uh every test will be executed
00:24:39.360
in parallel on all your CPU Um in 2019 I was Rubik Hagi talking
00:24:46.080
about the project called crystal ball which uh is a way to uh kind of keep the
00:24:52.640
track of coverage between the test and source code so that when you change the source code this tool would predict what
00:24:59.200
tests need to be run and as you've seen uh as long as the build and test graph
00:25:05.520
are properly maintained basil has it out of the box you run all the tests but
00:25:10.720
only the tests that are really needed to be rerun are being executed So you get
00:25:16.880
this behavior out of the box Um things that we kind of just had a quick glance
00:25:24.320
uh but every every package every test that you create in Basil has visibility
00:25:30.000
control you can say that this part of the codebase is private and maybe a
00:25:35.520
particular API is being publicly available and you can achieve the same behavior as with the tools like packwork
00:25:42.960
provide you but this works out of the box um with with
00:25:48.039
Basil and this whole thing with the caching this way when Basil keeps track
00:25:54.880
of all your inputs um it works great locally but it also So
00:26:00.559
can be used remotely So if you have hundreds of engineers and uh you can set
00:26:07.200
up a for example Google cloud storage some kind of distributed storage that
00:26:12.880
Basil will use automatically to share the results of tests and build between
00:26:18.320
all the engineers the CI system so that you know whenever somebody runs the
00:26:24.159
tests they actually don't run all of them They only run the tests that were
00:26:29.279
changed as they work on the software code And taking it further we can not
00:26:37.520
only share uh the cache we can also completely offload the execution of
00:26:43.840
these commands like RSpec gem build We can actually take all of that upload it
00:26:49.120
to a distributed cluster parallelize run things there and
00:26:54.400
then uh take download the results back to the computer machine So that's
00:27:00.559
literally uh I think every company big company that uses basil end up using
00:27:06.320
this thing called remote build execution where they have a cloud computing where
00:27:12.320
actually things are happening and developers just execute it from their machine with the same basil build
00:27:18.840
commands and uh what's currently done in rules Ruby is that this works with J
00:27:24.480
Ruby only and if there is anyone from C Ruby team Um I really ask you if you
00:27:32.240
could please create a standalone version of C Ruby that can be downloaded
00:27:37.360
portable and used without compiling and installing for specific architecture
00:27:43.440
uh specific computer because that currently limits this adoption of remote
00:27:48.720
build execution in rules Ruby And a couple of other things which I guess niche um is that rules Ruby allows you
00:27:57.200
to build the C code that depends on Ruby um headers and compile jars depending on
00:28:04.399
J Ruby classes This also is available in rules Ruby and there is still a lot to be done
00:28:11.679
because Basil is a huge project It has a lot of great things out of the box and
00:28:18.159
we are still working on adding some of those Um few things worth mentioning is
00:28:24.720
Basil has a idea for test sharding So when you run all the tests it cannot it
00:28:30.399
can only not not only parallelize it on your CPU but it can actually split it and shards and run on different uh
00:28:37.559
servers So if you use NAPSAC or that's gem that's kind of similar approach and
00:28:44.640
uh the test charting is not yet implemented with rules Ruby Basil has a
00:28:50.320
built-in support for coverage So when you run tests it would also tell you the coverage results for all your different
00:28:57.559
um not only Ruby but Java Python and we're still adding support for to have
00:29:03.919
it with simple cough Another great things in Basil called aspects which is
00:29:09.840
a way when you run the build when you run the test it would automatically execute extra commands on all the source
00:29:16.799
code Um so things like maybe type check the Ruby code during the build It's
00:29:22.480
something that we yet to support and a couple of other things which are Ruby
00:29:27.679
specific like the managing dependencies with gem file
00:29:33.039
is not 100% complete We don't yet support self-hosted Ruby gems or uh git
00:29:39.080
repositories and the debugging support is also it works but uh there is work to
00:29:44.399
be done there to make it more seamless uh with with with the debug
00:29:49.720
jam Um at this point I would like to thank you Please check out the
00:29:54.880
repository I hope I managed to show you at least that there is a different way to build and test the Ruby projects Uh
00:30:03.200
there is a basilu built slack where we have a ruby channel So if you start your
00:30:08.240
journey with basil and ruby it's a great place to ask questions or feel free to
00:30:14.080
send me an email And I would like to say special thank you to my new Ruby friend
00:30:21.360
uh Yamagami who actually I was standing in line for food truck when the food ran
00:30:26.880
out and he actually gave me his food So thank you very much for that And I I
00:30:32.080
guess he's starving now Thank you
