00:00:12.519
yeah Hello everyone Uh welcome to our
00:00:15.599
talk keeping secrets lessons learns
00:00:18.480
lessons learned from securing GitHub Uh
00:00:20.960
I'm Dennis from the security team at
00:00:22.720
GitHub
00:00:24.560
and I'm Wayin a security engineer at
00:00:26.640
Pritorian and we have come together to
00:00:28.640
collaborate on creating this talk um to
00:00:31.279
share the offensive and defensive side
00:00:33.520
of security Um so
00:00:36.920
yeah so if you're someone who is
00:00:39.600
interested in writing secure Ruby code
00:00:41.840
or finding vulnerabilities in your
00:00:43.680
existing Ruby code or just improving
00:00:46.079
your security posture as a whole then
00:00:48.239
this talk is for you And even if you're
00:00:50.559
relatively new to application security
00:00:52.960
uh I think the story that we'll be
00:00:54.239
telling here is a pretty compelling
00:00:56.320
reason to want to be more
00:00:58.520
involved So let's start with some quick
00:01:00.760
introductions Uh I'm Dennis My online
00:01:03.600
handle is Linux and uh I'm from Toronto
00:01:06.320
Canada Uh I work for GitHub which is
00:01:08.960
awesome because uh whether you're a
00:01:11.680
seasoned engineer writing critical
00:01:13.760
software or you're just starting off and
00:01:15.600
writing your first few lines of code
00:01:17.439
it's very likely that we get to be part
00:01:19.119
of your developer journey Uh so that's
00:01:21.840
great And before I was at GitHub I was
00:01:24.000
at the engineering team at Shopify Uh
00:01:26.400
which again I'm sure you're familiar
00:01:27.840
with given their contributions to the
00:01:29.520
Ruby community Uh including some of the
00:01:32.320
other talks like Maxim's talk next door
00:01:35.680
Uh the team I work on is uh at GitHub is
00:01:39.360
the product security engineering team
00:01:41.680
Our focus is on empowering our own
00:01:43.680
internal teams to consistently build and
00:01:46.079
maintain secure products and services
00:01:48.240
across
00:01:51.159
GitHub Hi everyone Um my name is Wayen
00:01:54.159
and I'm also known as Chris online and
00:01:56.640
I'm from Singapore Huge honor to be here
00:01:59.439
speaking to y'all today Um I currently
00:02:01.840
work for Pritorian Um we are a company
00:02:04.479
that provides offensive security
00:02:06.159
services to Fortune 500 and also other
00:02:08.720
large companies Um my team focuses on
00:02:11.920
product and application security where
00:02:14.480
we collaborate with our clients to help
00:02:16.720
them secure their products and
00:02:18.840
infrastructures Previously I was a
00:02:21.120
security researcher at Starab Singapore
00:02:23.520
where I was focusing a lot on web
00:02:25.680
security as well as security research
00:02:28.080
where I found this vulnerability that
00:02:29.840
we'll be talking a lot about today
00:02:34.000
So today Jennis and I have come together
00:02:36.480
to share some secret stories here at
00:02:38.800
Ruby Kaiigi Um I'll start off by sharing
00:02:42.640
um and bring you all along this
00:02:44.640
offensive security journey on how I
00:02:46.640
managed to hack GitHub by disclosing
00:02:49.280
their secrets and reporting to them
00:02:51.760
covering from the vulnerability
00:02:53.360
disclosure uh discovery process to
00:02:56.400
exploitation as well as um analyzing the
00:02:59.040
root cause of the security issues
00:03:03.440
Uh then I'm going to talk a bit about
00:03:05.519
the defensive security side of things Uh
00:03:08.239
covering what went on behind the scenes
00:03:09.920
at GitHub uh in responding to this
00:03:12.080
vulnerability as well as some of the
00:03:13.680
patterns that we as a Ruby community can
00:03:16.959
uh just utilize to better protect
00:03:18.920
ourselves And finally we'll end off with
00:03:21.280
a discussion of just how to handle
00:03:23.440
secrets in Ruby to ensure that well they
00:03:26.720
stay secret
00:03:31.680
So let's journey back to the day where I
00:03:33.920
found this vulnerability in GitHub Um
00:03:36.319
which is the day after Christmas in 2023
00:03:39.200
but technically it's still Christmas in
00:03:41.040
the US but well it's technically work
00:03:43.680
day for me So I was able to disclose
00:03:46.720
GitHub secrets and we're not just
00:03:48.879
talking about like getting hold of like
00:03:50.879
one or two secrets We're talking about
00:03:53.440
all the production secrets um of the
00:03:56.959
production container of github.com So
00:03:59.519
that's serrus Um I had access to things
00:04:02.720
like the commit signing keys I have
00:04:05.120
access to um long-term cloud environment
00:04:08.000
access keys So that's just crazy right
00:04:10.799
because it's unimaginable that you know
00:04:12.560
a company that is the size of GitHub um
00:04:16.160
they have like a super mature security
00:04:18.239
program and there are still bugs like
00:04:21.440
that that could just catch you caught
00:04:23.600
you off guard um
00:04:25.479
anytime of course it would be really
00:04:27.680
terrible if this was exploited by a
00:04:29.520
malicious attacker um so of course I
00:04:31.759
reach out immediately to them um and
00:04:34.160
they have to swiftly remediate it and
00:04:36.400
conduct a tower investigation to confirm
00:04:38.800
that it was not um exploited by
00:04:40.720
malicious actors in the past And yeah
00:04:43.759
sorry Dennis for working over the
00:04:45.680
Christmas and New Year
00:04:49.720
holidays So yeah before we dive into the
00:04:53.120
vulnerability details proper um I think
00:04:55.199
it's good to understand what even is
00:04:57.360
GitHub um the entire app itself So
00:05:00.600
github.com is just a huge Rails um
00:05:04.360
monolith And while github.com is closed
00:05:07.680
source well you can reverse engineer the
00:05:10.000
largely similar self-hosted version of
00:05:12.639
GitHub also known as the GitHub
00:05:14.560
enterprise server um to get most of the
00:05:17.120
source code Um and so yeah because it's
00:05:19.759
a Rails application it's going to follow
00:05:21.280
the MVC architecture And something
00:05:23.919
interesting and I guess relevant is that
00:05:26.720
apparently GitHub created the view
00:05:28.639
component framework and I find that
00:05:30.880
really amazing by the way Um so they use
00:05:32.960
that extensively within their codebase
00:05:34.800
to implement a component-driven user
00:05:36.800
interface where um components view
00:05:40.080
components in the form of Ruby objects
00:05:42.160
are being rendered as markup somewhat
00:05:44.160
like model
00:05:45.479
objects but for
00:05:47.800
views So here's the vulnerable piece of
00:05:50.960
code Um but yeah I just want to start
00:05:53.120
with you know um what's the whole
00:05:56.080
process like So after I extracted out
00:05:58.080
the source code from the um GitHub
00:06:00.160
enterprise server I started searching
00:06:02.000
for dangerous method invocations um and
00:06:04.560
I found this suspicious identifier
00:06:07.120
identifier for method here um in the
00:06:09.280
repository items view component So if
00:06:12.479
you notice like the identifier for
00:06:14.560
method is doing like a dynamic method
00:06:16.240
dispatch using send Um we have seen San
00:06:18.960
Bing mentioned quite a few times in Ruby
00:06:20.880
Kaii today um and and yesterday So um
00:06:23.840
that's something to talk about a lot Um
00:06:26.479
and of course yeah it's calling on the
00:06:28.400
repository identifier key instance
00:06:30.560
variable that has been um initialized
00:06:32.400
within the
00:06:34.520
constructor So if we trace the code
00:06:36.960
further you can see that the repository
00:06:39.199
items view component is being rendered
00:06:41.759
in the index method here of this
00:06:44.000
controller Um and the repository
00:06:46.639
identifier key is just simply pointing
00:06:49.120
to the parents uh ID key This means that
00:06:53.360
if we can reach this endpoint well we'll
00:06:56.160
be able to do any method um dispatch on
00:06:58.960
that repository
00:07:00.280
object But what does that mean And why
00:07:03.759
is it even insecure right So I think
00:07:06.639
it's important to even understand what
00:07:08.319
sand is because you know sand is very
00:07:11.520
powerful but there's not much talks
00:07:14.000
about what sand is under the hood So
00:07:16.160
sand is just a built-in method to do
00:07:18.720
dynamic method dispatch or also known as
00:07:21.599
reflection in other languages Um but
00:07:24.560
this is what you get um to to do meta
00:07:27.360
programming in Ruby So here we have just
00:07:29.919
a very simple hello world class where
00:07:32.400
you have like a print method um acing
00:07:34.960
some split arguments and it's just
00:07:36.400
printing out um as a string If you want
00:07:39.120
to invoke this method dynamically you
00:07:41.199
just pass the method name as a first
00:07:43.280
parameter and then any additional
00:07:45.840
arguments
00:07:48.280
afterwards Now let's just kind of
00:07:50.400
approach this with like a hacker's
00:07:52.000
mindset
00:07:53.479
Um if you have this piece of code and
00:07:56.240
you have two controllable user inputs
00:07:59.680
what would you do to exploit it Well
00:08:02.160
you'll probably just call eval to
00:08:04.160
execute arbitrary Ruby code or run shell
00:08:07.520
commands like ls
00:08:09.960
here And we can kind of just extrapolate
00:08:12.560
this if you have multiple controllable
00:08:14.479
arguments by just calling sand repeatly
00:08:17.120
And so it's going to be a massive
00:08:18.879
security nightmare if an attacker can
00:08:21.199
control um multiple arguments and invoke
00:08:23.680
arbitrary
00:08:25.160
methods So let's kind of just quickly
00:08:27.440
circle back to the potential unsafe um
00:08:30.400
vulnerability where we know that the
00:08:32.560
user controllable input um the
00:08:34.880
repository identifier key um is not
00:08:38.159
being validated Meaning we can just call
00:08:40.640
any methods using send um on the
00:08:43.120
repository object But it also looks like
00:08:45.839
the only control that we have is well
00:08:48.320
just a method name and we cannot pass in
00:08:50.160
any additional arguments like the
00:08:51.600
previous examples So this leads to like
00:08:54.160
a zero argument arbitrary method
00:08:56.880
invocation um constraint that we'll need
00:08:59.519
to um figure out in order to determine
00:09:01.839
whether this can be
00:09:04.440
exploited And so to to really um
00:09:07.839
understand this um what can we do to
00:09:10.399
deal with it Well we can just call any
00:09:12.800
methods that were defined on the on the
00:09:15.279
um repository class or those that were
00:09:18.000
inherited from their superasses Um in
00:09:20.640
fact that's why you can even call sand
00:09:22.160
in the first place right Because it's a
00:09:23.519
method from the kernel module and kernel
00:09:25.680
module is being included by object And
00:09:29.440
so um another thing that yeah there um
00:09:32.640
another thing to note here is that while
00:09:34.080
we cannot pass in additional arguments
00:09:36.720
this does not really restrict us to just
00:09:39.120
methods that don't accept arguments In
00:09:42.240
fact like these are just some list of
00:09:44.320
methods that you'll be able to call via
00:09:46.240
the unsafe reflection um so long as they
00:09:48.640
have erity of zero negative
00:09:51.560
one And on any Ruby given uh any given
00:09:55.519
Ruby object you'll be able to call this
00:09:57.680
inherited methods from the base classes
00:10:00.160
to leak some information um to aid in
00:10:03.279
exploiting the unvalidated send call Um
00:10:06.160
in my case well I have access to source
00:10:07.920
code so it doesn't really help me that
00:10:09.680
much Um but what would be useful to me
00:10:12.000
is that well there's a lot of functions
00:10:14.079
and I don't really want to trace them So
00:10:15.920
I just called the methods um method to
00:10:19.440
get a more context there on what
00:10:21.519
vulnerable functions there could be And
00:10:24.079
so what I did was just to drop into a
00:10:26.000
Rails console um in the GHS server um
00:10:29.440
and get a list of callable methods
00:10:33.160
Well interestingly enough the list I
00:10:35.839
ended up with had like 5,000 plus
00:10:38.000
methods initially and I pruned that down
00:10:40.160
to like 3.6,000 methods and most of it
00:10:42.959
is just crowd and and you know helper
00:10:45.519
methods that's being generated and
00:10:46.880
injected by Rails So yeah not too useful
00:10:50.800
Um but I really didn't want to prune the
00:10:53.120
entire list further So what I did was I
00:10:55.040
just kind of spread the entire list and
00:10:57.440
brute force every single meta name
00:10:59.760
against the vulnerable endpoint um and
00:11:02.640
just observing the responses and see if
00:11:04.560
I can leak anything interesting there Um
00:11:07.040
and what caught me off guard was that um
00:11:08.720
there were two endpoints that just
00:11:10.000
return a whole bunch of environment
00:11:11.640
variables And yeah I think you guys are
00:11:14.160
also confused right Like why is why is
00:11:16.399
there endpoints returning just or rather
00:11:19.120
two methods is returning a lot of M Um
00:11:22.880
so this definitely warranted additional
00:11:24.760
investigation and yeah sorry Dennis
00:11:27.600
since I kind of just disrupted your time
00:11:29.920
off work Um yeah I guess yeah I will
00:11:32.240
also just focus on like doing the root
00:11:35.120
cause analysis um just to help you yeah
00:11:38.240
get back to enjoying your
00:11:41.399
holidays Yeah So um this is one of the
00:11:44.399
method um NWFK is just network file
00:11:47.360
system check In fact I don't even know
00:11:49.519
what it is from a blackbox perspective
00:11:52.000
Um but this is a method found in g
00:11:54.560
dependency um of the repository um class
00:11:57.839
that is um that is included there So you
00:12:00.640
can see that it's just making an RPC
00:12:02.240
call to do um a spawn git um meta call
00:12:06.800
and eventually the code just flows down
00:12:08.880
to this spawn method um and returns this
00:12:12.279
hash Although the nth here in the
00:12:14.959
argument um is set to empty hash the nth
00:12:18.240
global variable is actually being passed
00:12:19.839
to spawn So we get all the environmental
00:12:22.480
variables that's being passed to this
00:12:24.800
rails application in the return value
00:12:28.399
I have no idea why you do that to be
00:12:30.240
honest Um that's probably an oversight
00:12:33.360
So yeah now that we've determined the
00:12:35.440
root cause of this vulnerability is that
00:12:37.279
really all that we can do Well not quite
00:12:40.160
Um there is still one more thing we can
00:12:42.160
do which is we can actually chain that
00:12:44.480
unvalidated senor to get remote code
00:12:47.120
execution on GitHub enterprise servers
00:12:49.760
using the GH vendor cookie So the J
00:12:52.800
vendor cookie defaults to using Marshall
00:12:55.040
for serializing session data that has
00:12:57.200
been signed and encrypted using the
00:12:59.279
enterprise session secret environment
00:13:01.360
variable which we can leak So yeah I
00:13:05.600
just want to do a quick shout out to
00:13:06.800
Samuel Giddens for pointing out and and
00:13:09.040
Ruby Kai last year Marshall was never
00:13:11.600
really designed with security in mind So
00:13:14.320
if an attacker can dialize arbitrary
00:13:16.800
payloads using Marshall they're going to
00:13:19.519
end up with remote code execution on
00:13:21.360
your
00:13:22.680
systems So now that we've seen like how
00:13:25.200
damaging potentially damaging this
00:13:26.800
vulnerability can be well let's switch
00:13:29.279
it up to the defensive side of um of
00:13:31.680
security and have Dennis share his
00:13:33.959
experience And yeah hopefully Dennis
00:13:36.720
doesn't come after me for interrupting
00:13:37.920
his time off work
00:13:40.639
Thanks Wayin Well yeah To be honest I
00:13:43.760
was hoping to disconnect from work
00:13:45.600
during the Christmas holiday but hey
00:13:48.000
security never sleeps right So in this
00:13:51.360
next section I'll go over what
00:13:53.200
responding to this vulnerabil
00:13:54.639
vulnerability looked like within GitHub
00:13:56.800
and what secure patterns were
00:13:58.399
implemented to patch things up Also give
00:14:01.040
some practical advice about some tools
00:14:02.880
you may want to adopt to protect your
00:14:04.880
Ruby code
00:14:07.040
All right So to help understand what the
00:14:08.959
response uh to this vulnerability looked
00:14:11.360
like let's step through what the
00:14:12.720
vulnerability life cycle looks like at
00:14:15.160
GitHub The first step in the process is
00:14:17.440
how we intake new vulnerabilities In
00:14:19.680
this case the discovery came from the
00:14:21.440
bug bounty program but it's worth noting
00:14:23.600
that we also ingest large number of
00:14:25.680
vulnerability alerts from our automated
00:14:27.440
code scanning tools And we also get
00:14:29.760
reports from our internal security or
00:14:32.160
engineering teams and sometimes even
00:14:34.079
external sources like our
00:14:36.279
customers The next step is triage We
00:14:39.199
analyze the vulnerability assess the
00:14:41.199
risk associated with it and prioritize
00:14:43.600
it depending on how quickly our teams
00:14:45.680
need to respond In this case Whan's
00:14:48.160
discovery was assessed as a critical
00:14:50.160
vulnerability with broad impact So we
00:14:53.120
need to respond and begin remediation as
00:14:55.440
soon as possible
00:14:57.600
At this point a bunch of pagers went off
00:14:59.839
and uh our incident response team pulled
00:15:01.680
in a bunch of relevant people from
00:15:03.440
different teams across GitHub so that we
00:15:05.360
could individually understand the scope
00:15:07.120
of the impact and begin remediation This
00:15:09.839
whole step in the life cycle really
00:15:11.440
involves multiple phases First
00:15:13.760
containment and eradication ensuring
00:15:15.519
that we can contain and cut off any
00:15:17.199
actors that might be in the process of
00:15:18.800
exploiting this vulnerability In this
00:15:20.880
case based on our investigation we uh
00:15:23.519
didn't have any evidence that there was
00:15:24.800
additional exploitation beside Whan's
00:15:27.040
report So we were good in that regard
00:15:29.360
But uh and then there's the mitigation
00:15:31.440
and remediation step which involves the
00:15:33.600
immediate fixes of blocking the
00:15:35.440
vulnerability and also longerterm post
00:15:37.920
incident response research security
00:15:40.560
hardening all that stuff that we do But
00:15:43.279
for now let's look at what some of the
00:15:44.880
code changes that we implemented were
00:15:47.519
We've seen how powerful send can be So
00:15:49.759
our first fix was to address its usage
00:15:51.920
in the identifier for method Luckily as
00:15:55.040
it turns out this code wasn't actually
00:15:56.720
strictly dependent on using send but
00:15:59.040
rather it was likely written that way as
00:16:00.800
a convenience due to the assumption that
00:16:02.720
the repository identifier key would
00:16:04.880
either be a valid ID key or fall back to
00:16:07.600
the global relay ID provided in the
00:16:10.839
constructor So the fix here is pretty
00:16:13.040
straightforward it's just to directly
00:16:14.800
call the global relay ID method if the
00:16:17.120
value isn't specifically our other type
00:16:19.680
of valid ID Uh as a side note I also
00:16:22.639
just want to mention that we operate in
00:16:24.160
a blameless culture at GitHub So
00:16:26.639
whenever we're going through a
00:16:27.759
vulnerability report or bug report like
00:16:30.160
this uh once we've addressed the
00:16:31.839
immediate risk we then turn our focus
00:16:33.680
back to figuring out how we can all
00:16:35.120
collectively learn uh and improve from
00:16:37.360
this event without pointing fingers at
00:16:39.120
anyone
00:16:40.959
Anyway the question you might be asking
00:16:42.560
now is what if I am in a situation where
00:16:45.120
I need the flexibility that send gives
00:16:47.040
me Can I use it safely Uh well it turns
00:16:50.320
out that there is an object method you
00:16:52.240
can use called public send which as you
00:16:54.800
might guess uh can only call public
00:16:57.120
methods on your target
00:16:59.160
object But remember that send itself is
00:17:03.199
a public method on every object So we
00:17:05.439
can call send using public send and then
00:17:07.919
call any method we want Uh this does
00:17:10.640
prevent zero and singlear argument
00:17:12.400
arbitrary method invocation which is
00:17:14.079
great Uh but if an attacker were to
00:17:16.400
control multiple arguments then they
00:17:18.640
could still do a lot of damage with
00:17:20.000
public
00:17:21.240
send The safest way to use send is to
00:17:24.079
pass a static set of values as callable
00:17:26.240
methods Or alternatively you might even
00:17:29.039
want to call the same method across
00:17:30.720
multiple targets But again you just need
00:17:33.440
to make sure that those targets are
00:17:34.720
controlled by you and not directly
00:17:36.240
accepting user inputs The caveat with
00:17:39.039
both of these approaches is that you
00:17:41.039
also then need to make sure that all of
00:17:42.640
the methods that you're calling can
00:17:44.160
potentially handle unsafe additional
00:17:46.360
arguments Uh so yeah if you're using
00:17:49.039
send you're cooking with fire Uh you
00:17:51.120
just need to be extra
00:17:52.679
careful Now back to patching our code Uh
00:17:55.840
additionally here if you remember from
00:17:58.000
earlier the uh controller that was
00:17:59.840
rendering the repository item component
00:18:02.320
we can improve the way the RID key
00:18:04.000
method works by simply guarding against
00:18:06.240
unexpected inputs This alone would have
00:18:09.120
been enough to prevent the vulnerability
00:18:11.440
It just goes to show that it's the
00:18:13.280
little things it's the small cracks in
00:18:15.039
our codebase that as our as the
00:18:17.120
complexity of our applications grow
00:18:18.960
those are the things that will be
00:18:20.240
eventually chained together and
00:18:21.840
exploited by attackers
00:18:25.039
uh we also scope down the environment
00:18:26.640
variables that were passed to that get
00:18:28.480
RPC process Uh so rather than passing in
00:18:31.520
that n global hash uh which contains all
00:18:34.320
the sensitive environment variables
00:18:35.760
we're now passing in a very small subset
00:18:37.840
of environment variables to that process
00:18:40.160
that it depends on Um we also don't even
00:18:42.880
return this hash anymore This
00:18:44.559
screenshot's a bit out of date but yeah
00:18:46.480
the thing is you may be tempted to make
00:18:48.400
copies of env uh or pass it around
00:18:51.200
different parts of your codebase Uh it's
00:18:53.440
easy to forget that for most Ruby web
00:18:55.840
apps getting access to env means
00:18:58.080
accessing a whole bunch of sensitive
00:18:59.600
data like the keys to the city Uh so you
00:19:02.640
need to treat this somewhat unassuming
00:19:04.720
hash with the same level of care that
00:19:06.240
you would users passwords for
00:19:09.080
example And finally regarding that
00:19:11.360
vulnerable cookie that Whan mentioned uh
00:19:14.400
yeah that that was using Marshall uh at
00:19:17.120
GitHub we actually have had already
00:19:18.960
moved away from using Marshall in favor
00:19:20.799
of things like JSON serialization but
00:19:23.520
that cookie was somehow left over Uh we
00:19:26.080
were able to entirely remove that cookie
00:19:28.160
since it was used for a service that had
00:19:29.919
already been deprecated and was no
00:19:31.679
longer being used Again it's the little
00:19:34.000
things that you need to make sure to
00:19:36.000
clean up Okay back on track the
00:19:39.360
vulnerable code path had been patched
00:19:41.280
but in terms of our immediate response
00:19:43.200
efforts we still needed to rotate all of
00:19:45.840
the exposed secrets Uh this is bes
00:19:48.799
because despite coming from a trusted
00:19:50.799
source through our bug bounty program uh
00:19:53.280
we still needed to treat all of these
00:19:54.880
secrets as being fully compromised Any
00:19:57.120
exposure outside of our protected
00:19:58.880
systems is technically unauthorized
00:20:01.480
access But we'll we'll eventually come
00:20:03.760
back to this in the final chapter Let's
00:20:05.919
wrap up the code security side of the
00:20:07.600
story first
00:20:09.360
in terms of ensuring that our code isn't
00:20:10.960
vulnerable to similar attacks Uh at the
00:20:12.960
same time that remediation is happening
00:20:14.640
we also kick off a process called
00:20:16.080
variant analysis Uh if you're not
00:20:18.080
familiar with this term it basically
00:20:19.600
means that we're taking this
00:20:20.720
vulnerability as a starting point for
00:20:22.720
modeling the the vulnerability and
00:20:24.799
expanding on it to try and detect
00:20:26.640
variance of it that might also be
00:20:28.720
present in our code Uh unfortunately I
00:20:30.960
don't have too much time to get into how
00:20:32.559
this works but we'll have some resources
00:20:34.559
linked at the end of the talk if you're
00:20:35.919
interested in this kind of thing
00:20:38.559
And finally we could disclose the
00:20:40.240
incident publicly Uh yeah we deployed
00:20:44.480
several types of user notifications
00:20:46.159
direct emails public blog post uh
00:20:48.799
issuing a CVE a common vulnerability
00:20:51.520
enumeration for GHS and releasing new
00:20:54.640
patches for yeah GitHub enterprise
00:20:58.520
server Okay great We did it That's
00:21:01.919
awesome Uh but you may be looking at
00:21:04.320
this process and think yeah that's
00:21:06.000
that's nice GitHub has a huge security
00:21:07.840
team with lots of resources at its
00:21:09.840
disposal Uh what about my startup or my
00:21:12.240
like open source
00:21:14.440
project Well your organization doesn't
00:21:16.799
need to be the size of GitHub to benefit
00:21:18.880
from some of the code scanning tools
00:21:20.240
that we have out there For example we
00:21:22.320
have Breakman for broadly detecting
00:21:24.400
vulnerabil vulnerable code in Rails apps
00:21:27.200
Uh if you're not doing any code scanning
00:21:28.720
already this one's really easy to just
00:21:30.400
start using at any stage of your
00:21:31.919
development Uh rubocop is obviously
00:21:34.880
another popular linting tool um that
00:21:37.679
you're probably familiar with And
00:21:39.120
there's there's actually a lot of
00:21:40.159
security focused cops out there that
00:21:42.400
just in the community that can prevent
00:21:44.000
risky Ruby code from being merged in the
00:21:45.760
f in the first place Uh for example our
00:21:48.000
friends over at GitLab security um
00:21:50.320
published one called public send that
00:21:52.320
blocks any usage of send or public
00:21:54.840
send These tools do have their
00:21:56.880
limitations though since dynamic code
00:21:58.559
can be very difficult to analyze So you
00:22:00.720
may need to try different tools to
00:22:02.080
achieve the types of detections that
00:22:03.440
you're looking for Um for example we can
00:22:06.799
look at some more sophisticated tools
00:22:08.400
like uh Smra OpenGP uh using a
00:22:11.120
combination of a parsing and fuzzy
00:22:13.200
matching to get more accurate results
00:22:15.520
And of course there's GitHub's favorite
00:22:17.120
our very own CodeQL Uh CodeQL is free
00:22:20.480
for research and open source projects
00:22:22.320
Otherwise it is a paid product as part
00:22:24.480
of our uh GitHub code security offerings
00:22:27.120
But it's a really powerful tool for
00:22:29.120
generating detections with a really high
00:22:30.799
degree of accuracy meaning very few
00:22:32.960
false positives and also just useful for
00:22:35.360
doing investigative security research
00:22:37.360
and variant analysis things like that
00:22:40.320
And to tie this all back the reason why
00:22:42.320
our CodeQL scan didn't catch the or our
00:22:45.039
CodeQL scans didn't catch the
00:22:46.559
vulnerability that Whan discovered was
00:22:48.799
because there was a gap in our query set
00:22:51.360
which wasn't able to fully traverse how
00:22:53.760
those view components were being
00:22:55.360
rendered in this case But as a result of
00:22:58.320
this we updated uh the default query set
00:23:01.520
So now everyone in the community using
00:23:03.520
CodeQL got to benefit from this
00:23:05.200
discovery
00:23:08.080
So some takeaways use powerful language
00:23:10.880
features with great
00:23:12.200
care Consider the trade-offs and
00:23:14.559
security implications of using features
00:23:16.559
like send marshall metarogramming
00:23:19.600
features that we have Um utilize code
00:23:22.480
scanning tools to uncover
00:23:23.919
vulnerabilities in your code before they
00:23:25.919
can be exploited Uh and even include
00:23:28.559
linting to prevent dangerous code from
00:23:30.320
even being shipped in the first place
00:23:32.480
And of course use guard statements
00:23:34.960
validate or sanitize user controlled
00:23:37.120
inputs Just because you've passed CI
00:23:39.760
you've passed your SC code scanning
00:23:41.520
everything looks good doesn't mean that
00:23:43.280
your code is guaranteed to be
00:23:46.039
secure Now our final chapter keeping
00:23:49.280
secrets the catchy tagline of this talk
00:23:52.159
uh where we talk about the whole
00:23:53.679
pipeline of managing secrets in your
00:23:55.919
Ruby app
00:23:57.679
So probably the biggest takeaway for
00:23:59.200
GitHub from this exposure event was
00:24:00.960
tackling the challenges involved in
00:24:02.480
rotating so many secrets at once Uh
00:24:05.200
first when you look at the names of
00:24:07.039
environment variables that have been
00:24:08.480
exposed it's it can be hard to tell if
00:24:10.559
it's a secret or if it's just
00:24:12.279
configuration So taking inventory there
00:24:14.799
is important Uh if the value that you
00:24:17.360
found is a secret then identifying which
00:24:19.840
teams own that secret uh can be
00:24:22.240
surprisingly hard Uh ideally we should
00:24:24.880
also be able to automate the rotation of
00:24:26.720
secrets which we do Um but it's
00:24:29.440
difficult in certain cases especially
00:24:30.960
with things like encryption keys and
00:24:32.720
signing keys and things that produce
00:24:34.559
longived artifacts in your app Uh you
00:24:37.039
can't just swap these encryption keys
00:24:38.960
out without also employing some kind of
00:24:40.960
like multi key strategy where you keep
00:24:42.880
the old decryption key around while you
00:24:44.640
phase it out
00:24:46.279
Um yeah just a thoughtprovoking question
00:24:49.039
for you is like do you know how long it
00:24:51.200
would take for you to rotate all of your
00:24:53.039
production secrets It's not something we
00:24:55.279
think about very often until you know
00:24:57.279
until we have
00:24:58.440
to This stuff isn't easy but the main
00:25:00.799
takeaway here is to have a playbook and
00:25:03.440
actually run through that playbook to
00:25:05.200
make sure that you can rotate things
00:25:06.640
swiftly because that'll drastically
00:25:09.279
reduce the impact of an exposure
00:25:12.679
event As for the storage and management
00:25:15.039
of secrets there's obviously multiple
00:25:17.840
ways you could do this You could start
00:25:19.200
with an end file like a M file Don't
00:25:22.400
commit this to GitHub though Please
00:25:24.240
don't do that Very bad Uh but this
00:25:26.880
doesn't scale Uh and you could look at
00:25:29.600
something like Rails's encrypted
00:25:31.279
credentials to safely encrypt and then
00:25:34.080
commit the secrets to your repo But this
00:25:36.640
also isn't great because then you're
00:25:38.159
still dependent on a single encryption
00:25:40.000
key and key management is hard The best
00:25:43.039
option we have is something like a
00:25:44.480
network secret store So like Hashi
00:25:46.720
Cororp vault Azure Key Vault AWS Vault
00:25:48.960
I'm pretty sure every cloud provider has
00:25:50.720
some version of this Uh the big benefits
00:25:53.200
of this being that you get auditability
00:25:56.159
You get just in time scoped access to
00:25:58.240
your secrets Uh we can apply the concept
00:26:00.799
of lease privilege uh for how we're
00:26:02.960
using these secrets And you can easily
00:26:05.600
manage the versions of these secrets uh
00:26:08.080
which will be very useful now that
00:26:09.679
everyone here will be rotating all of
00:26:11.039
their secrets all the
00:26:13.960
time And finally a question I've asked
00:26:16.640
myself uh after this incident is is it
00:26:19.360
even possible to protect sensitive
00:26:21.440
values in Ruby if someone can act uh
00:26:23.840
execute arbitrary code in your runtime
00:26:26.559
Uh is there like some can we be clever
00:26:29.200
and introduce some kind of friction to
00:26:30.960
slow down attackers Well yeah this is
00:26:34.240
tricky because we could do things like
00:26:35.919
overloading methods that are commonly
00:26:37.600
used by attackers We could overload
00:26:40.000
instance variable get or like the
00:26:41.760
methods method uh to slow them down but
00:26:44.960
this just comes at the trade-off of
00:26:47.279
slowing ourselves down and making our
00:26:49.039
own systems harder to debug So uh
00:26:52.080
generally probably not worth the trouble
00:26:54.880
I think the real goal here is to
00:26:56.720
minimize the memory footprint of your
00:26:58.400
sensitive data Uh so something we can do
00:27:01.200
is to not store our secrets in that
00:27:03.520
global environment hash env uh or
00:27:06.720
initially load them into env but then
00:27:08.880
delete them once the application is
00:27:10.679
initialized Um even just loading your
00:27:13.360
secrets from env into a custom class
00:27:16.320
allows you to be more intentional about
00:27:18.480
how you're managing this sensitive data
00:27:21.520
um and how your developers are
00:27:23.279
interacting with it potentially
00:27:24.799
providing a better development
00:27:26.480
experience or doing some more
00:27:28.760
sophisticated fetching You can do a lot
00:27:31.039
of things here but it's something to
00:27:33.039
think
00:27:34.760
about Uh so yeah that's uh that's kind
00:27:38.159
of our thoughts here Uh we also have a
00:27:40.400
lot of resources If you're interested in
00:27:42.159
any of the words that were spoken during
00:27:44.159
this talk uh there's a QR code and links
00:27:47.760
to additional resources Um frack is a
00:27:50.720
really good starting point for just
00:27:52.480
finding like looking into finding
00:27:54.240
vulnerability in Ruby and Rails apps Um
00:27:58.399
CodeQL is like super powerful tool
00:28:01.760
There's a lot of resources about how to
00:28:03.120
get started with that and learn how to
00:28:04.559
use it And uh also bonus shout out to my
00:28:07.679
teammates uh Kylie and Matt for their
00:28:09.919
Railscom talk which uh talks about how
00:28:13.039
we implemented active record encryption
00:28:15.840
at GitHub uh kind of tangentially
00:28:18.720
related to this but another security
00:28:20.720
topic that you might be interested in
00:28:24.960
Yeah And of course I have to give like a
00:28:27.840
quick shout out to GitHub security team
00:28:29.600
for running an amazing bug bounty
00:28:31.279
program um that facilitates the
00:28:33.600
reporting of security vulnerabilities in
00:28:35.440
GitHub And so if you are interested to
00:28:37.679
learn more yeah please do visit
00:28:40.440
bounty.github.com And also like this
00:28:42.640
this monarch head is extremely cute Um I
00:28:45.520
initially thought it was undocumented
00:28:47.120
but apparently it's investig
00:28:49.440
investigator cat or something like that
00:28:52.399
Yeah interesting stuff So yeah that's it
00:28:56.720
Um if you have anything um that you want
00:28:59.440
to ask us yeah feel free to come chat
00:29:01.039
with us Yeah thank you