Summarized using AI

Keeping Secrets: Lessons Learned From Securing GitHub

Dennis Pacewicz and Wei Lin Ngo • April 17, 2025 • Matsuyama, Ehime, Japan • Talk

In their talk "Keeping Secrets: Lessons Learned From Securing GitHub" at RubyKaigi 2025, speakers Dennis Pacewicz and Wei Lin Ngo explore critical lessons learned from a significant security incident involving GitHub. They delve into the dual facets of security—offensive and defensive—providing crucial insights for developers, particularly those working with Ruby.

Key Points Discussed:

  • Introduction to Speakers: Dennis from GitHub and Wei from Pritorian introduce themselves, highlighting their backgrounds in application security and vulnerability assessment.

  • Offensive Security Journey: Wei recounts a personal experience of discovering a major vulnerability in GitHub right after Christmas 2023, which granted him access to sensitive production secrets, including commit signing keys and cloud access keys. He details the discovery process, the exploitation methods used, and the importance of promptly reporting vulnerabilities to responsible teams.

  • Vulnerability Details: Wei explains the nature of the vulnerability related to the misuse of the send method in Ruby, which could potentially allow attackers to execute arbitrary code. He emphasizes the risks associated with unvalidated user inputs and illustrates how he exploited this weakness to reveal environment variables.

  • Defensive Mitigation: Dennis outlines GitHub's response strategy to vulnerabilities reported through their bug bounty program. This includes triage, containment, and remediation efforts, with a specific focus on the necessity of cross-team collaboration during incident response. He provides advice on practical code changes made as a result of the vulnerability, emphasizing a blameless culture in their team environment.

  • Code Security Tools and Recommendations: The speakers advocate for the use of various code scanning tools such as Brakeman, RuboCop, and CodeQL to identify vulnerabilities proactively before they can be exploited. They stress the importance of guarding against dangerous code practices and ensuring sound input validation.

  • Managing Secrets in Ruby: The talk concludes with a discussion on best practices for managing secrets in Ruby applications. They recommend using secret management tools and processes to handle sensitive data securely and emphasize routine secret rotation as a preventive measure against potential leaks.

Key Takeaways:

  • Maintain rigorous security practices, especially when utilizing powerful Ruby features.
  • Adopt security scanning tools to uncover and mitigate vulnerabilities early in the development process.
  • Educate developers on the importance of validating inputs and securely managing application secrets to safeguard applications against potential breaches.

Keeping Secrets: Lessons Learned From Securing GitHub
Dennis Pacewicz and Wei Lin Ngo • Matsuyama, Ehime, Japan • Talk

Date: April 17, 2025
Published: May 27, 2025
Announced: unknown

RubyKaigi 2025

00:00:12.519 yeah Hello everyone Uh welcome to our
00:00:15.599 talk keeping secrets lessons learns
00:00:18.480 lessons learned from securing GitHub Uh
00:00:20.960 I'm Dennis from the security team at
00:00:22.720 GitHub
00:00:24.560 and I'm Wayin a security engineer at
00:00:26.640 Pritorian and we have come together to
00:00:28.640 collaborate on creating this talk um to
00:00:31.279 share the offensive and defensive side
00:00:33.520 of security Um so
00:00:36.920 yeah so if you're someone who is
00:00:39.600 interested in writing secure Ruby code
00:00:41.840 or finding vulnerabilities in your
00:00:43.680 existing Ruby code or just improving
00:00:46.079 your security posture as a whole then
00:00:48.239 this talk is for you And even if you're
00:00:50.559 relatively new to application security
00:00:52.960 uh I think the story that we'll be
00:00:54.239 telling here is a pretty compelling
00:00:56.320 reason to want to be more
00:00:58.520 involved So let's start with some quick
00:01:00.760 introductions Uh I'm Dennis My online
00:01:03.600 handle is Linux and uh I'm from Toronto
00:01:06.320 Canada Uh I work for GitHub which is
00:01:08.960 awesome because uh whether you're a
00:01:11.680 seasoned engineer writing critical
00:01:13.760 software or you're just starting off and
00:01:15.600 writing your first few lines of code
00:01:17.439 it's very likely that we get to be part
00:01:19.119 of your developer journey Uh so that's
00:01:21.840 great And before I was at GitHub I was
00:01:24.000 at the engineering team at Shopify Uh
00:01:26.400 which again I'm sure you're familiar
00:01:27.840 with given their contributions to the
00:01:29.520 Ruby community Uh including some of the
00:01:32.320 other talks like Maxim's talk next door
00:01:35.680 Uh the team I work on is uh at GitHub is
00:01:39.360 the product security engineering team
00:01:41.680 Our focus is on empowering our own
00:01:43.680 internal teams to consistently build and
00:01:46.079 maintain secure products and services
00:01:48.240 across
00:01:51.159 GitHub Hi everyone Um my name is Wayen
00:01:54.159 and I'm also known as Chris online and
00:01:56.640 I'm from Singapore Huge honor to be here
00:01:59.439 speaking to y'all today Um I currently
00:02:01.840 work for Pritorian Um we are a company
00:02:04.479 that provides offensive security
00:02:06.159 services to Fortune 500 and also other
00:02:08.720 large companies Um my team focuses on
00:02:11.920 product and application security where
00:02:14.480 we collaborate with our clients to help
00:02:16.720 them secure their products and
00:02:18.840 infrastructures Previously I was a
00:02:21.120 security researcher at Starab Singapore
00:02:23.520 where I was focusing a lot on web
00:02:25.680 security as well as security research
00:02:28.080 where I found this vulnerability that
00:02:29.840 we'll be talking a lot about today
00:02:34.000 So today Jennis and I have come together
00:02:36.480 to share some secret stories here at
00:02:38.800 Ruby Kaiigi Um I'll start off by sharing
00:02:42.640 um and bring you all along this
00:02:44.640 offensive security journey on how I
00:02:46.640 managed to hack GitHub by disclosing
00:02:49.280 their secrets and reporting to them
00:02:51.760 covering from the vulnerability
00:02:53.360 disclosure uh discovery process to
00:02:56.400 exploitation as well as um analyzing the
00:02:59.040 root cause of the security issues
00:03:03.440 Uh then I'm going to talk a bit about
00:03:05.519 the defensive security side of things Uh
00:03:08.239 covering what went on behind the scenes
00:03:09.920 at GitHub uh in responding to this
00:03:12.080 vulnerability as well as some of the
00:03:13.680 patterns that we as a Ruby community can
00:03:16.959 uh just utilize to better protect
00:03:18.920 ourselves And finally we'll end off with
00:03:21.280 a discussion of just how to handle
00:03:23.440 secrets in Ruby to ensure that well they
00:03:26.720 stay secret
00:03:31.680 So let's journey back to the day where I
00:03:33.920 found this vulnerability in GitHub Um
00:03:36.319 which is the day after Christmas in 2023
00:03:39.200 but technically it's still Christmas in
00:03:41.040 the US but well it's technically work
00:03:43.680 day for me So I was able to disclose
00:03:46.720 GitHub secrets and we're not just
00:03:48.879 talking about like getting hold of like
00:03:50.879 one or two secrets We're talking about
00:03:53.440 all the production secrets um of the
00:03:56.959 production container of github.com So
00:03:59.519 that's serrus Um I had access to things
00:04:02.720 like the commit signing keys I have
00:04:05.120 access to um long-term cloud environment
00:04:08.000 access keys So that's just crazy right
00:04:10.799 because it's unimaginable that you know
00:04:12.560 a company that is the size of GitHub um
00:04:16.160 they have like a super mature security
00:04:18.239 program and there are still bugs like
00:04:21.440 that that could just catch you caught
00:04:23.600 you off guard um
00:04:25.479 anytime of course it would be really
00:04:27.680 terrible if this was exploited by a
00:04:29.520 malicious attacker um so of course I
00:04:31.759 reach out immediately to them um and
00:04:34.160 they have to swiftly remediate it and
00:04:36.400 conduct a tower investigation to confirm
00:04:38.800 that it was not um exploited by
00:04:40.720 malicious actors in the past And yeah
00:04:43.759 sorry Dennis for working over the
00:04:45.680 Christmas and New Year
00:04:49.720 holidays So yeah before we dive into the
00:04:53.120 vulnerability details proper um I think
00:04:55.199 it's good to understand what even is
00:04:57.360 GitHub um the entire app itself So
00:05:00.600 github.com is just a huge Rails um
00:05:04.360 monolith And while github.com is closed
00:05:07.680 source well you can reverse engineer the
00:05:10.000 largely similar self-hosted version of
00:05:12.639 GitHub also known as the GitHub
00:05:14.560 enterprise server um to get most of the
00:05:17.120 source code Um and so yeah because it's
00:05:19.759 a Rails application it's going to follow
00:05:21.280 the MVC architecture And something
00:05:23.919 interesting and I guess relevant is that
00:05:26.720 apparently GitHub created the view
00:05:28.639 component framework and I find that
00:05:30.880 really amazing by the way Um so they use
00:05:32.960 that extensively within their codebase
00:05:34.800 to implement a component-driven user
00:05:36.800 interface where um components view
00:05:40.080 components in the form of Ruby objects
00:05:42.160 are being rendered as markup somewhat
00:05:44.160 like model
00:05:45.479 objects but for
00:05:47.800 views So here's the vulnerable piece of
00:05:50.960 code Um but yeah I just want to start
00:05:53.120 with you know um what's the whole
00:05:56.080 process like So after I extracted out
00:05:58.080 the source code from the um GitHub
00:06:00.160 enterprise server I started searching
00:06:02.000 for dangerous method invocations um and
00:06:04.560 I found this suspicious identifier
00:06:07.120 identifier for method here um in the
00:06:09.280 repository items view component So if
00:06:12.479 you notice like the identifier for
00:06:14.560 method is doing like a dynamic method
00:06:16.240 dispatch using send Um we have seen San
00:06:18.960 Bing mentioned quite a few times in Ruby
00:06:20.880 Kaii today um and and yesterday So um
00:06:23.840 that's something to talk about a lot Um
00:06:26.479 and of course yeah it's calling on the
00:06:28.400 repository identifier key instance
00:06:30.560 variable that has been um initialized
00:06:32.400 within the
00:06:34.520 constructor So if we trace the code
00:06:36.960 further you can see that the repository
00:06:39.199 items view component is being rendered
00:06:41.759 in the index method here of this
00:06:44.000 controller Um and the repository
00:06:46.639 identifier key is just simply pointing
00:06:49.120 to the parents uh ID key This means that
00:06:53.360 if we can reach this endpoint well we'll
00:06:56.160 be able to do any method um dispatch on
00:06:58.960 that repository
00:07:00.280 object But what does that mean And why
00:07:03.759 is it even insecure right So I think
00:07:06.639 it's important to even understand what
00:07:08.319 sand is because you know sand is very
00:07:11.520 powerful but there's not much talks
00:07:14.000 about what sand is under the hood So
00:07:16.160 sand is just a built-in method to do
00:07:18.720 dynamic method dispatch or also known as
00:07:21.599 reflection in other languages Um but
00:07:24.560 this is what you get um to to do meta
00:07:27.360 programming in Ruby So here we have just
00:07:29.919 a very simple hello world class where
00:07:32.400 you have like a print method um acing
00:07:34.960 some split arguments and it's just
00:07:36.400 printing out um as a string If you want
00:07:39.120 to invoke this method dynamically you
00:07:41.199 just pass the method name as a first
00:07:43.280 parameter and then any additional
00:07:45.840 arguments
00:07:48.280 afterwards Now let's just kind of
00:07:50.400 approach this with like a hacker's
00:07:52.000 mindset
00:07:53.479 Um if you have this piece of code and
00:07:56.240 you have two controllable user inputs
00:07:59.680 what would you do to exploit it Well
00:08:02.160 you'll probably just call eval to
00:08:04.160 execute arbitrary Ruby code or run shell
00:08:07.520 commands like ls
00:08:09.960 here And we can kind of just extrapolate
00:08:12.560 this if you have multiple controllable
00:08:14.479 arguments by just calling sand repeatly
00:08:17.120 And so it's going to be a massive
00:08:18.879 security nightmare if an attacker can
00:08:21.199 control um multiple arguments and invoke
00:08:23.680 arbitrary
00:08:25.160 methods So let's kind of just quickly
00:08:27.440 circle back to the potential unsafe um
00:08:30.400 vulnerability where we know that the
00:08:32.560 user controllable input um the
00:08:34.880 repository identifier key um is not
00:08:38.159 being validated Meaning we can just call
00:08:40.640 any methods using send um on the
00:08:43.120 repository object But it also looks like
00:08:45.839 the only control that we have is well
00:08:48.320 just a method name and we cannot pass in
00:08:50.160 any additional arguments like the
00:08:51.600 previous examples So this leads to like
00:08:54.160 a zero argument arbitrary method
00:08:56.880 invocation um constraint that we'll need
00:08:59.519 to um figure out in order to determine
00:09:01.839 whether this can be
00:09:04.440 exploited And so to to really um
00:09:07.839 understand this um what can we do to
00:09:10.399 deal with it Well we can just call any
00:09:12.800 methods that were defined on the on the
00:09:15.279 um repository class or those that were
00:09:18.000 inherited from their superasses Um in
00:09:20.640 fact that's why you can even call sand
00:09:22.160 in the first place right Because it's a
00:09:23.519 method from the kernel module and kernel
00:09:25.680 module is being included by object And
00:09:29.440 so um another thing that yeah there um
00:09:32.640 another thing to note here is that while
00:09:34.080 we cannot pass in additional arguments
00:09:36.720 this does not really restrict us to just
00:09:39.120 methods that don't accept arguments In
00:09:42.240 fact like these are just some list of
00:09:44.320 methods that you'll be able to call via
00:09:46.240 the unsafe reflection um so long as they
00:09:48.640 have erity of zero negative
00:09:51.560 one And on any Ruby given uh any given
00:09:55.519 Ruby object you'll be able to call this
00:09:57.680 inherited methods from the base classes
00:10:00.160 to leak some information um to aid in
00:10:03.279 exploiting the unvalidated send call Um
00:10:06.160 in my case well I have access to source
00:10:07.920 code so it doesn't really help me that
00:10:09.680 much Um but what would be useful to me
00:10:12.000 is that well there's a lot of functions
00:10:14.079 and I don't really want to trace them So
00:10:15.920 I just called the methods um method to
00:10:19.440 get a more context there on what
00:10:21.519 vulnerable functions there could be And
00:10:24.079 so what I did was just to drop into a
00:10:26.000 Rails console um in the GHS server um
00:10:29.440 and get a list of callable methods
00:10:33.160 Well interestingly enough the list I
00:10:35.839 ended up with had like 5,000 plus
00:10:38.000 methods initially and I pruned that down
00:10:40.160 to like 3.6,000 methods and most of it
00:10:42.959 is just crowd and and you know helper
00:10:45.519 methods that's being generated and
00:10:46.880 injected by Rails So yeah not too useful
00:10:50.800 Um but I really didn't want to prune the
00:10:53.120 entire list further So what I did was I
00:10:55.040 just kind of spread the entire list and
00:10:57.440 brute force every single meta name
00:10:59.760 against the vulnerable endpoint um and
00:11:02.640 just observing the responses and see if
00:11:04.560 I can leak anything interesting there Um
00:11:07.040 and what caught me off guard was that um
00:11:08.720 there were two endpoints that just
00:11:10.000 return a whole bunch of environment
00:11:11.640 variables And yeah I think you guys are
00:11:14.160 also confused right Like why is why is
00:11:16.399 there endpoints returning just or rather
00:11:19.120 two methods is returning a lot of M Um
00:11:22.880 so this definitely warranted additional
00:11:24.760 investigation and yeah sorry Dennis
00:11:27.600 since I kind of just disrupted your time
00:11:29.920 off work Um yeah I guess yeah I will
00:11:32.240 also just focus on like doing the root
00:11:35.120 cause analysis um just to help you yeah
00:11:38.240 get back to enjoying your
00:11:41.399 holidays Yeah So um this is one of the
00:11:44.399 method um NWFK is just network file
00:11:47.360 system check In fact I don't even know
00:11:49.519 what it is from a blackbox perspective
00:11:52.000 Um but this is a method found in g
00:11:54.560 dependency um of the repository um class
00:11:57.839 that is um that is included there So you
00:12:00.640 can see that it's just making an RPC
00:12:02.240 call to do um a spawn git um meta call
00:12:06.800 and eventually the code just flows down
00:12:08.880 to this spawn method um and returns this
00:12:12.279 hash Although the nth here in the
00:12:14.959 argument um is set to empty hash the nth
00:12:18.240 global variable is actually being passed
00:12:19.839 to spawn So we get all the environmental
00:12:22.480 variables that's being passed to this
00:12:24.800 rails application in the return value
00:12:28.399 I have no idea why you do that to be
00:12:30.240 honest Um that's probably an oversight
00:12:33.360 So yeah now that we've determined the
00:12:35.440 root cause of this vulnerability is that
00:12:37.279 really all that we can do Well not quite
00:12:40.160 Um there is still one more thing we can
00:12:42.160 do which is we can actually chain that
00:12:44.480 unvalidated senor to get remote code
00:12:47.120 execution on GitHub enterprise servers
00:12:49.760 using the GH vendor cookie So the J
00:12:52.800 vendor cookie defaults to using Marshall
00:12:55.040 for serializing session data that has
00:12:57.200 been signed and encrypted using the
00:12:59.279 enterprise session secret environment
00:13:01.360 variable which we can leak So yeah I
00:13:05.600 just want to do a quick shout out to
00:13:06.800 Samuel Giddens for pointing out and and
00:13:09.040 Ruby Kai last year Marshall was never
00:13:11.600 really designed with security in mind So
00:13:14.320 if an attacker can dialize arbitrary
00:13:16.800 payloads using Marshall they're going to
00:13:19.519 end up with remote code execution on
00:13:21.360 your
00:13:22.680 systems So now that we've seen like how
00:13:25.200 damaging potentially damaging this
00:13:26.800 vulnerability can be well let's switch
00:13:29.279 it up to the defensive side of um of
00:13:31.680 security and have Dennis share his
00:13:33.959 experience And yeah hopefully Dennis
00:13:36.720 doesn't come after me for interrupting
00:13:37.920 his time off work
00:13:40.639 Thanks Wayin Well yeah To be honest I
00:13:43.760 was hoping to disconnect from work
00:13:45.600 during the Christmas holiday but hey
00:13:48.000 security never sleeps right So in this
00:13:51.360 next section I'll go over what
00:13:53.200 responding to this vulnerabil
00:13:54.639 vulnerability looked like within GitHub
00:13:56.800 and what secure patterns were
00:13:58.399 implemented to patch things up Also give
00:14:01.040 some practical advice about some tools
00:14:02.880 you may want to adopt to protect your
00:14:04.880 Ruby code
00:14:07.040 All right So to help understand what the
00:14:08.959 response uh to this vulnerability looked
00:14:11.360 like let's step through what the
00:14:12.720 vulnerability life cycle looks like at
00:14:15.160 GitHub The first step in the process is
00:14:17.440 how we intake new vulnerabilities In
00:14:19.680 this case the discovery came from the
00:14:21.440 bug bounty program but it's worth noting
00:14:23.600 that we also ingest large number of
00:14:25.680 vulnerability alerts from our automated
00:14:27.440 code scanning tools And we also get
00:14:29.760 reports from our internal security or
00:14:32.160 engineering teams and sometimes even
00:14:34.079 external sources like our
00:14:36.279 customers The next step is triage We
00:14:39.199 analyze the vulnerability assess the
00:14:41.199 risk associated with it and prioritize
00:14:43.600 it depending on how quickly our teams
00:14:45.680 need to respond In this case Whan's
00:14:48.160 discovery was assessed as a critical
00:14:50.160 vulnerability with broad impact So we
00:14:53.120 need to respond and begin remediation as
00:14:55.440 soon as possible
00:14:57.600 At this point a bunch of pagers went off
00:14:59.839 and uh our incident response team pulled
00:15:01.680 in a bunch of relevant people from
00:15:03.440 different teams across GitHub so that we
00:15:05.360 could individually understand the scope
00:15:07.120 of the impact and begin remediation This
00:15:09.839 whole step in the life cycle really
00:15:11.440 involves multiple phases First
00:15:13.760 containment and eradication ensuring
00:15:15.519 that we can contain and cut off any
00:15:17.199 actors that might be in the process of
00:15:18.800 exploiting this vulnerability In this
00:15:20.880 case based on our investigation we uh
00:15:23.519 didn't have any evidence that there was
00:15:24.800 additional exploitation beside Whan's
00:15:27.040 report So we were good in that regard
00:15:29.360 But uh and then there's the mitigation
00:15:31.440 and remediation step which involves the
00:15:33.600 immediate fixes of blocking the
00:15:35.440 vulnerability and also longerterm post
00:15:37.920 incident response research security
00:15:40.560 hardening all that stuff that we do But
00:15:43.279 for now let's look at what some of the
00:15:44.880 code changes that we implemented were
00:15:47.519 We've seen how powerful send can be So
00:15:49.759 our first fix was to address its usage
00:15:51.920 in the identifier for method Luckily as
00:15:55.040 it turns out this code wasn't actually
00:15:56.720 strictly dependent on using send but
00:15:59.040 rather it was likely written that way as
00:16:00.800 a convenience due to the assumption that
00:16:02.720 the repository identifier key would
00:16:04.880 either be a valid ID key or fall back to
00:16:07.600 the global relay ID provided in the
00:16:10.839 constructor So the fix here is pretty
00:16:13.040 straightforward it's just to directly
00:16:14.800 call the global relay ID method if the
00:16:17.120 value isn't specifically our other type
00:16:19.680 of valid ID Uh as a side note I also
00:16:22.639 just want to mention that we operate in
00:16:24.160 a blameless culture at GitHub So
00:16:26.639 whenever we're going through a
00:16:27.759 vulnerability report or bug report like
00:16:30.160 this uh once we've addressed the
00:16:31.839 immediate risk we then turn our focus
00:16:33.680 back to figuring out how we can all
00:16:35.120 collectively learn uh and improve from
00:16:37.360 this event without pointing fingers at
00:16:39.120 anyone
00:16:40.959 Anyway the question you might be asking
00:16:42.560 now is what if I am in a situation where
00:16:45.120 I need the flexibility that send gives
00:16:47.040 me Can I use it safely Uh well it turns
00:16:50.320 out that there is an object method you
00:16:52.240 can use called public send which as you
00:16:54.800 might guess uh can only call public
00:16:57.120 methods on your target
00:16:59.160 object But remember that send itself is
00:17:03.199 a public method on every object So we
00:17:05.439 can call send using public send and then
00:17:07.919 call any method we want Uh this does
00:17:10.640 prevent zero and singlear argument
00:17:12.400 arbitrary method invocation which is
00:17:14.079 great Uh but if an attacker were to
00:17:16.400 control multiple arguments then they
00:17:18.640 could still do a lot of damage with
00:17:20.000 public
00:17:21.240 send The safest way to use send is to
00:17:24.079 pass a static set of values as callable
00:17:26.240 methods Or alternatively you might even
00:17:29.039 want to call the same method across
00:17:30.720 multiple targets But again you just need
00:17:33.440 to make sure that those targets are
00:17:34.720 controlled by you and not directly
00:17:36.240 accepting user inputs The caveat with
00:17:39.039 both of these approaches is that you
00:17:41.039 also then need to make sure that all of
00:17:42.640 the methods that you're calling can
00:17:44.160 potentially handle unsafe additional
00:17:46.360 arguments Uh so yeah if you're using
00:17:49.039 send you're cooking with fire Uh you
00:17:51.120 just need to be extra
00:17:52.679 careful Now back to patching our code Uh
00:17:55.840 additionally here if you remember from
00:17:58.000 earlier the uh controller that was
00:17:59.840 rendering the repository item component
00:18:02.320 we can improve the way the RID key
00:18:04.000 method works by simply guarding against
00:18:06.240 unexpected inputs This alone would have
00:18:09.120 been enough to prevent the vulnerability
00:18:11.440 It just goes to show that it's the
00:18:13.280 little things it's the small cracks in
00:18:15.039 our codebase that as our as the
00:18:17.120 complexity of our applications grow
00:18:18.960 those are the things that will be
00:18:20.240 eventually chained together and
00:18:21.840 exploited by attackers
00:18:25.039 uh we also scope down the environment
00:18:26.640 variables that were passed to that get
00:18:28.480 RPC process Uh so rather than passing in
00:18:31.520 that n global hash uh which contains all
00:18:34.320 the sensitive environment variables
00:18:35.760 we're now passing in a very small subset
00:18:37.840 of environment variables to that process
00:18:40.160 that it depends on Um we also don't even
00:18:42.880 return this hash anymore This
00:18:44.559 screenshot's a bit out of date but yeah
00:18:46.480 the thing is you may be tempted to make
00:18:48.400 copies of env uh or pass it around
00:18:51.200 different parts of your codebase Uh it's
00:18:53.440 easy to forget that for most Ruby web
00:18:55.840 apps getting access to env means
00:18:58.080 accessing a whole bunch of sensitive
00:18:59.600 data like the keys to the city Uh so you
00:19:02.640 need to treat this somewhat unassuming
00:19:04.720 hash with the same level of care that
00:19:06.240 you would users passwords for
00:19:09.080 example And finally regarding that
00:19:11.360 vulnerable cookie that Whan mentioned uh
00:19:14.400 yeah that that was using Marshall uh at
00:19:17.120 GitHub we actually have had already
00:19:18.960 moved away from using Marshall in favor
00:19:20.799 of things like JSON serialization but
00:19:23.520 that cookie was somehow left over Uh we
00:19:26.080 were able to entirely remove that cookie
00:19:28.160 since it was used for a service that had
00:19:29.919 already been deprecated and was no
00:19:31.679 longer being used Again it's the little
00:19:34.000 things that you need to make sure to
00:19:36.000 clean up Okay back on track the
00:19:39.360 vulnerable code path had been patched
00:19:41.280 but in terms of our immediate response
00:19:43.200 efforts we still needed to rotate all of
00:19:45.840 the exposed secrets Uh this is bes
00:19:48.799 because despite coming from a trusted
00:19:50.799 source through our bug bounty program uh
00:19:53.280 we still needed to treat all of these
00:19:54.880 secrets as being fully compromised Any
00:19:57.120 exposure outside of our protected
00:19:58.880 systems is technically unauthorized
00:20:01.480 access But we'll we'll eventually come
00:20:03.760 back to this in the final chapter Let's
00:20:05.919 wrap up the code security side of the
00:20:07.600 story first
00:20:09.360 in terms of ensuring that our code isn't
00:20:10.960 vulnerable to similar attacks Uh at the
00:20:12.960 same time that remediation is happening
00:20:14.640 we also kick off a process called
00:20:16.080 variant analysis Uh if you're not
00:20:18.080 familiar with this term it basically
00:20:19.600 means that we're taking this
00:20:20.720 vulnerability as a starting point for
00:20:22.720 modeling the the vulnerability and
00:20:24.799 expanding on it to try and detect
00:20:26.640 variance of it that might also be
00:20:28.720 present in our code Uh unfortunately I
00:20:30.960 don't have too much time to get into how
00:20:32.559 this works but we'll have some resources
00:20:34.559 linked at the end of the talk if you're
00:20:35.919 interested in this kind of thing
00:20:38.559 And finally we could disclose the
00:20:40.240 incident publicly Uh yeah we deployed
00:20:44.480 several types of user notifications
00:20:46.159 direct emails public blog post uh
00:20:48.799 issuing a CVE a common vulnerability
00:20:51.520 enumeration for GHS and releasing new
00:20:54.640 patches for yeah GitHub enterprise
00:20:58.520 server Okay great We did it That's
00:21:01.919 awesome Uh but you may be looking at
00:21:04.320 this process and think yeah that's
00:21:06.000 that's nice GitHub has a huge security
00:21:07.840 team with lots of resources at its
00:21:09.840 disposal Uh what about my startup or my
00:21:12.240 like open source
00:21:14.440 project Well your organization doesn't
00:21:16.799 need to be the size of GitHub to benefit
00:21:18.880 from some of the code scanning tools
00:21:20.240 that we have out there For example we
00:21:22.320 have Breakman for broadly detecting
00:21:24.400 vulnerabil vulnerable code in Rails apps
00:21:27.200 Uh if you're not doing any code scanning
00:21:28.720 already this one's really easy to just
00:21:30.400 start using at any stage of your
00:21:31.919 development Uh rubocop is obviously
00:21:34.880 another popular linting tool um that
00:21:37.679 you're probably familiar with And
00:21:39.120 there's there's actually a lot of
00:21:40.159 security focused cops out there that
00:21:42.400 just in the community that can prevent
00:21:44.000 risky Ruby code from being merged in the
00:21:45.760 f in the first place Uh for example our
00:21:48.000 friends over at GitLab security um
00:21:50.320 published one called public send that
00:21:52.320 blocks any usage of send or public
00:21:54.840 send These tools do have their
00:21:56.880 limitations though since dynamic code
00:21:58.559 can be very difficult to analyze So you
00:22:00.720 may need to try different tools to
00:22:02.080 achieve the types of detections that
00:22:03.440 you're looking for Um for example we can
00:22:06.799 look at some more sophisticated tools
00:22:08.400 like uh Smra OpenGP uh using a
00:22:11.120 combination of a parsing and fuzzy
00:22:13.200 matching to get more accurate results
00:22:15.520 And of course there's GitHub's favorite
00:22:17.120 our very own CodeQL Uh CodeQL is free
00:22:20.480 for research and open source projects
00:22:22.320 Otherwise it is a paid product as part
00:22:24.480 of our uh GitHub code security offerings
00:22:27.120 But it's a really powerful tool for
00:22:29.120 generating detections with a really high
00:22:30.799 degree of accuracy meaning very few
00:22:32.960 false positives and also just useful for
00:22:35.360 doing investigative security research
00:22:37.360 and variant analysis things like that
00:22:40.320 And to tie this all back the reason why
00:22:42.320 our CodeQL scan didn't catch the or our
00:22:45.039 CodeQL scans didn't catch the
00:22:46.559 vulnerability that Whan discovered was
00:22:48.799 because there was a gap in our query set
00:22:51.360 which wasn't able to fully traverse how
00:22:53.760 those view components were being
00:22:55.360 rendered in this case But as a result of
00:22:58.320 this we updated uh the default query set
00:23:01.520 So now everyone in the community using
00:23:03.520 CodeQL got to benefit from this
00:23:05.200 discovery
00:23:08.080 So some takeaways use powerful language
00:23:10.880 features with great
00:23:12.200 care Consider the trade-offs and
00:23:14.559 security implications of using features
00:23:16.559 like send marshall metarogramming
00:23:19.600 features that we have Um utilize code
00:23:22.480 scanning tools to uncover
00:23:23.919 vulnerabilities in your code before they
00:23:25.919 can be exploited Uh and even include
00:23:28.559 linting to prevent dangerous code from
00:23:30.320 even being shipped in the first place
00:23:32.480 And of course use guard statements
00:23:34.960 validate or sanitize user controlled
00:23:37.120 inputs Just because you've passed CI
00:23:39.760 you've passed your SC code scanning
00:23:41.520 everything looks good doesn't mean that
00:23:43.280 your code is guaranteed to be
00:23:46.039 secure Now our final chapter keeping
00:23:49.280 secrets the catchy tagline of this talk
00:23:52.159 uh where we talk about the whole
00:23:53.679 pipeline of managing secrets in your
00:23:55.919 Ruby app
00:23:57.679 So probably the biggest takeaway for
00:23:59.200 GitHub from this exposure event was
00:24:00.960 tackling the challenges involved in
00:24:02.480 rotating so many secrets at once Uh
00:24:05.200 first when you look at the names of
00:24:07.039 environment variables that have been
00:24:08.480 exposed it's it can be hard to tell if
00:24:10.559 it's a secret or if it's just
00:24:12.279 configuration So taking inventory there
00:24:14.799 is important Uh if the value that you
00:24:17.360 found is a secret then identifying which
00:24:19.840 teams own that secret uh can be
00:24:22.240 surprisingly hard Uh ideally we should
00:24:24.880 also be able to automate the rotation of
00:24:26.720 secrets which we do Um but it's
00:24:29.440 difficult in certain cases especially
00:24:30.960 with things like encryption keys and
00:24:32.720 signing keys and things that produce
00:24:34.559 longived artifacts in your app Uh you
00:24:37.039 can't just swap these encryption keys
00:24:38.960 out without also employing some kind of
00:24:40.960 like multi key strategy where you keep
00:24:42.880 the old decryption key around while you
00:24:44.640 phase it out
00:24:46.279 Um yeah just a thoughtprovoking question
00:24:49.039 for you is like do you know how long it
00:24:51.200 would take for you to rotate all of your
00:24:53.039 production secrets It's not something we
00:24:55.279 think about very often until you know
00:24:57.279 until we have
00:24:58.440 to This stuff isn't easy but the main
00:25:00.799 takeaway here is to have a playbook and
00:25:03.440 actually run through that playbook to
00:25:05.200 make sure that you can rotate things
00:25:06.640 swiftly because that'll drastically
00:25:09.279 reduce the impact of an exposure
00:25:12.679 event As for the storage and management
00:25:15.039 of secrets there's obviously multiple
00:25:17.840 ways you could do this You could start
00:25:19.200 with an end file like a M file Don't
00:25:22.400 commit this to GitHub though Please
00:25:24.240 don't do that Very bad Uh but this
00:25:26.880 doesn't scale Uh and you could look at
00:25:29.600 something like Rails's encrypted
00:25:31.279 credentials to safely encrypt and then
00:25:34.080 commit the secrets to your repo But this
00:25:36.640 also isn't great because then you're
00:25:38.159 still dependent on a single encryption
00:25:40.000 key and key management is hard The best
00:25:43.039 option we have is something like a
00:25:44.480 network secret store So like Hashi
00:25:46.720 Cororp vault Azure Key Vault AWS Vault
00:25:48.960 I'm pretty sure every cloud provider has
00:25:50.720 some version of this Uh the big benefits
00:25:53.200 of this being that you get auditability
00:25:56.159 You get just in time scoped access to
00:25:58.240 your secrets Uh we can apply the concept
00:26:00.799 of lease privilege uh for how we're
00:26:02.960 using these secrets And you can easily
00:26:05.600 manage the versions of these secrets uh
00:26:08.080 which will be very useful now that
00:26:09.679 everyone here will be rotating all of
00:26:11.039 their secrets all the
00:26:13.960 time And finally a question I've asked
00:26:16.640 myself uh after this incident is is it
00:26:19.360 even possible to protect sensitive
00:26:21.440 values in Ruby if someone can act uh
00:26:23.840 execute arbitrary code in your runtime
00:26:26.559 Uh is there like some can we be clever
00:26:29.200 and introduce some kind of friction to
00:26:30.960 slow down attackers Well yeah this is
00:26:34.240 tricky because we could do things like
00:26:35.919 overloading methods that are commonly
00:26:37.600 used by attackers We could overload
00:26:40.000 instance variable get or like the
00:26:41.760 methods method uh to slow them down but
00:26:44.960 this just comes at the trade-off of
00:26:47.279 slowing ourselves down and making our
00:26:49.039 own systems harder to debug So uh
00:26:52.080 generally probably not worth the trouble
00:26:54.880 I think the real goal here is to
00:26:56.720 minimize the memory footprint of your
00:26:58.400 sensitive data Uh so something we can do
00:27:01.200 is to not store our secrets in that
00:27:03.520 global environment hash env uh or
00:27:06.720 initially load them into env but then
00:27:08.880 delete them once the application is
00:27:10.679 initialized Um even just loading your
00:27:13.360 secrets from env into a custom class
00:27:16.320 allows you to be more intentional about
00:27:18.480 how you're managing this sensitive data
00:27:21.520 um and how your developers are
00:27:23.279 interacting with it potentially
00:27:24.799 providing a better development
00:27:26.480 experience or doing some more
00:27:28.760 sophisticated fetching You can do a lot
00:27:31.039 of things here but it's something to
00:27:33.039 think
00:27:34.760 about Uh so yeah that's uh that's kind
00:27:38.159 of our thoughts here Uh we also have a
00:27:40.400 lot of resources If you're interested in
00:27:42.159 any of the words that were spoken during
00:27:44.159 this talk uh there's a QR code and links
00:27:47.760 to additional resources Um frack is a
00:27:50.720 really good starting point for just
00:27:52.480 finding like looking into finding
00:27:54.240 vulnerability in Ruby and Rails apps Um
00:27:58.399 CodeQL is like super powerful tool
00:28:01.760 There's a lot of resources about how to
00:28:03.120 get started with that and learn how to
00:28:04.559 use it And uh also bonus shout out to my
00:28:07.679 teammates uh Kylie and Matt for their
00:28:09.919 Railscom talk which uh talks about how
00:28:13.039 we implemented active record encryption
00:28:15.840 at GitHub uh kind of tangentially
00:28:18.720 related to this but another security
00:28:20.720 topic that you might be interested in
00:28:24.960 Yeah And of course I have to give like a
00:28:27.840 quick shout out to GitHub security team
00:28:29.600 for running an amazing bug bounty
00:28:31.279 program um that facilitates the
00:28:33.600 reporting of security vulnerabilities in
00:28:35.440 GitHub And so if you are interested to
00:28:37.679 learn more yeah please do visit
00:28:40.440 bounty.github.com And also like this
00:28:42.640 this monarch head is extremely cute Um I
00:28:45.520 initially thought it was undocumented
00:28:47.120 but apparently it's investig
00:28:49.440 investigator cat or something like that
00:28:52.399 Yeah interesting stuff So yeah that's it
00:28:56.720 Um if you have anything um that you want
00:28:59.440 to ask us yeah feel free to come chat
00:29:01.039 with us Yeah thank you
Explore all talks recorded at RubyKaigi 2025
+66