50.000 processed records per second: a CRuby & JRuby story


Summarized using AI

50.000 processed records per second: a CRuby & JRuby story

Cristian Planas • April 16, 2025 • Matsuyama, Ehime, Japan • Talk

The talk titled "50,000 processed records per second: a CRuby & JRuby story" by Cristian Planas presented at RubyKaigi 2025 discusses the development and scaling of the Zendesk Indexer, a microservice responsible for indexing vast quantities of records into ElasticSearch. This service initially began as a series of scripts in C-Ruby and evolved into a comprehensive application using JRuby, motivated by the need for faster record updates and better scalability.

Key Points Discussed:
- Introduction to the Speaker: Cristian Planas introduces himself and his extensive experience with Ruby, emphasizing his passion for scalability issues in Ruby applications.
- Historical Context of Ruby: The speaker reflects on the popularity of Ruby and Ruby on Rails during their peak, portraying an idyllic view of the Ruby programming community and its technological advancements.
- Challenges of Concurrency and Scalability: Issues such as the Global Interpreter Lock (GIL) and memory management in Ruby are presented as significant challenges, but Planas explains that developments in JRuby and concurrency support have enabled better solutions.
- The ETL Pattern at Zendesk: The talk outlines the ETL (Extract, Transform, Load) process used in the Zendesk Indexer, detailing how data is handled from various sources including MySQL and DynamoDB before being loaded into ElasticSearch.
- Migration to JRuby: The transition to JRuby allowed for true multithreading capabilities and access to Java libraries, which played a critical role in the scaling of the service. JRuby's performance enabled Zendesk to handle around 50,000 updates in one second regularly.
- Learning from Production Issues: The talk covers several lessons learned from maintaining the Indexer, including coding style unification between Ruby and Java, dealing with library differences, and ensuring thread safety across various libraries.
- Future of Ruby and Scalability: Planas discusses how modern tools like Kubernetes have changed the landscape for scalability, reducing dependency on threading issues. He emphasizes that many scalability concerns discussed in the past may no longer apply to current practices.

Conclusions and Takeaways:
- The evolution of Ruby, particularly with JRuby, showcases significant improvements in performance and concurrency management.
- Building microservices does not always equate to better scalability; sometimes, monolithic architectures can be more beneficial with the right tools.
- The Ruby community continues to innovate, with a focus on improving the performance and concurrency of Ruby applications.
- Engineering productivity and choosing the right tool for the team remain critical for successful software development, reinforcing the value of Ruby environments conducive to programmer happiness.

50.000 processed records per second: a CRuby & JRuby story
Cristian Planas • Matsuyama, Ehime, Japan • Talk

Date: April 16, 2025
Published: May 27, 2025
Announced: unknown

In this talk, we will share the fascinating journey of the Zendesk Indexer, a microservice responsible for indexing tens of thousands of records per second from a relational database into ElasticSearch. Originally born from a need to speed up record updates, the Indexer evolved from a series of scripts written in C-Ruby into a full application built in JRuby. Over the years, it incorporated diverse technologies like Riak and S3 and tackled significant challenges, particularly with concurrency, as Zendesk scaled. Now, we are preparing to reintegrate it into the Rails monolith. Attendees will learn valuable lessons about concurrency, safety mechanisms and scaling.

https://rubykaigi.org/2025/presentations/cristian_planas.html

RubyKaigi 2025

00:00:06.240 hello uh my name is Christian uh welcome
00:00:08.720 to my talk So this talk is called uh
00:00:11.280 50,000 process records per second a Ruby
00:00:14.000 and J Ruby story But first of all let me
00:00:17.039 uh let me say hello Hello hello Japan
00:00:19.920 And now I will I will introduce myself
00:00:22.720 Most important thing about me I would
00:00:24.640 say is that uh I am from Barcelona I
00:00:27.920 don't know if you've ever been there I
00:00:30.320 really recommend it Uh it's an amazing
00:00:33.360 city Uh the second most important thing
00:00:36.160 and probably the reason why I'm here is
00:00:38.079 that I've been doing Ruby for now 15
00:00:40.079 years This is a picture from the old
00:00:42.719 times You know uh you know match looks
00:00:45.280 great still I look very different Um but
00:00:48.399 in those times in in the 2010s I created
00:00:51.200 a company called Playful Bet I was the
00:00:53.760 chief technical officer which in that
00:00:56.160 context meant that I was the only
00:00:57.920 engineer there Um and I had to make it
00:01:01.199 scale No it was pretty successful Uh was
00:01:04.640 a great a huge challenge Uh but a
00:01:07.119 certain point I decided to hey I should
00:01:09.360 maybe try to go to a bigger company and
00:01:11.360 learn you know how to how to work how to
00:01:14.479 make big Ruby applications And I joined
00:01:17.840 Zenesk I've been there for more than a
00:01:19.680 decade and I currently work on
00:01:22.159 scalability issues This actually it's a
00:01:25.600 almost a personal thing for me like
00:01:27.680 these ideas like you know Rubies is slow
00:01:30.080 or rail doesn't scale it it really
00:01:32.960 really makes me angry you know it's like
00:01:34.400 something I I think all the time about
00:01:36.720 and actually this is the reason why I
00:01:38.479 start doing public speaking I well I do
00:01:41.759 I go around the world I do a lot of
00:01:44.000 talks about scalability issues I put a
00:01:46.560 lot of anime memes uh and I eventually
00:01:50.240 wrote a book uh the book is going to be
00:01:52.479 published um next month Uh but you
00:01:55.680 already can buy the beta if you want and
00:01:58.000 I got you a
00:01:59.640 discount But let's go let's go back to
00:02:02.320 to the topic of today This story is not
00:02:04.560 going to I'm not going to talk about
00:02:05.759 Rails I know I mean Rubik
00:02:08.319 uh we're going to talk about Ruby and
00:02:10.319 specifically why um companies choose
00:02:13.920 Ruby to write services or not But for
00:02:17.599 this topic we are going to do a little
00:02:19.840 bit of time traveling
00:02:22.480 We're going to go back to 15 20 years
00:02:27.000 ago And actually I think that this this
00:02:29.680 time traveling is going to be kind of
00:02:31.200 welcome because you know let's be honest
00:02:33.440 it's it's kind of hard times for us
00:02:35.840 developers You are in Twitter X whatever
00:02:38.640 blue sky You don't stop listening that
00:02:41.360 you know we're going to disappear It's
00:02:42.879 the last years of software engineers as
00:02:44.959 a as a profession We're going to be
00:02:47.200 replaced by some kind of AIS
00:02:51.120 Still I would say one thing If you've
00:02:53.519 been in the profession for a while this
00:02:56.000 is not the first time that you've been
00:02:57.360 listening this thing Uh I've been doing
00:02:59.840 this for 15 20 years and I've been in
00:03:03.040 part I've heard or I've been through
00:03:05.760 times in which we were also going to be
00:03:07.680 replaced when I was studying in in in
00:03:10.720 college I remember that I had a
00:03:12.800 professor that told me that programming
00:03:14.560 was useless because the future was UML
00:03:17.040 we would just draw boxes and and that
00:03:19.599 would speed the code That that that was
00:03:21.480 it That didn't happen Still it's still
00:03:26.159 hard times now for the profession There
00:03:28.800 were there were better times I think For
00:03:31.519 example 15 years ago for Ruby
00:03:33.920 particularly 15 20 years ago we were in
00:03:36.319 the in the peak of the of the hype This
00:03:38.480 is like the Garner hype cycle It was
00:03:40.640 incredibly popular I have them have very
00:03:44.080 different memories of those times I I
00:03:46.159 guess that if you were doing Ruby at
00:03:48.080 that time you you you you also had them
00:03:50.959 And well in those times uh Ruby and also
00:03:53.519 Rails powered a revolution a
00:03:55.920 technological
00:03:57.480 revolution You know for me those times
00:04:00.159 and and and that moment in time for Ruby
00:04:02.239 I imagine it as some kind of idyllic
00:04:04.159 oldtime village I'm going to show you
00:04:05.760 some some pictures of how that village
00:04:07.680 was No I imagine like this kind of nice
00:04:11.519 town in maybe Washington Oregon very
00:04:13.920 fresh nature This is
00:04:16.840 me and you know the city is called
00:04:20.440 Ruvita You know it's like this typical
00:04:23.120 Midwestern uh uh American town You have
00:04:26.320 the
00:04:27.320 cafe and in that cafe you find familiar
00:04:30.240 characters that make your life better
00:04:32.800 For example you have readable syntax I I
00:04:36.720 am of the generation that I learned to
00:04:38.560 program doing C and then if you got
00:04:41.280 lucky you did some Java and it was this
00:04:43.120 time that Java was incredly boiler plate
00:04:45.680 everywhere it was very hard you start
00:04:47.600 doing Ruby and that's so readable but at
00:04:50.880 the same time Ruby was in incredly
00:04:53.240 expressive with very few lines of code
00:04:55.520 you could express so
00:04:57.240 much it was
00:04:59.400 great and thanks to all of this uh um a
00:05:03.440 lot of very very powerful very very
00:05:05.600 popular uh companies were able to grow I
00:05:08.800 think see companies like Shopify GitHub
00:05:12.000 Twitter originally
00:05:14.199 Zenesk and these new planetary scale
00:05:17.080 organizations had new kinds of
00:05:20.120 needs They were able to horizontally
00:05:22.800 scale their their web applications
00:05:25.039 through you know using some
00:05:26.600 architectural architectural techniques
00:05:29.360 like for example
00:05:32.039 caching Another one was sharding So it
00:05:35.919 seemed like the thing was they were able
00:05:37.840 to scale but there was something wrong
00:05:41.440 in this idyllic ruby town We're going to
00:05:44.639 discuss it uh with an example We're
00:05:46.639 going to discuss it with an example of a
00:05:48.800 micros service that we have at Zenesk
00:05:51.520 that is it follows the ETL
00:05:54.680 pattern Okay What's the ETL here well I
00:05:58.320 like to say that we software engineers
00:05:59.840 we are basically data plumbers We we
00:06:02.479 move data from point to point from
00:06:04.319 application to application from cluster
00:06:06.080 to cluster An ETL is a pattern that does
00:06:08.960 exactly
00:06:10.280 that How does it work well it's divided
00:06:14.000 into into three steps In the first one
00:06:16.720 you extract data from a data source You
00:06:19.759 you fetch you fetch it In our case we
00:06:22.000 had to fetch data from MySQL from React
00:06:25.600 and from Dynamob
00:06:29.440 In the in the next step you transform it
00:06:31.600 You typically are going to want to
00:06:33.120 sanitize all that data to to merge it
00:06:36.639 you know to transform it You give it a
00:06:38.960 new structure You change the data types
00:06:40.800 the duplicate
00:06:43.080 etc And then eventually you load it into
00:06:45.759 the to the destination data source In
00:06:49.440 our case the destination data source was
00:06:51.360 elastic search Part of the
00:06:53.120 transformation was also detecting the
00:06:55.600 language because you know we wanted to
00:06:57.280 be able to run textual search on it Here
00:07:00.800 the question is like how many TL systems
00:07:03.280 are written in Ruby Well my guess is
00:07:05.840 that not many but we decided that we
00:07:08.560 wanted to keep being a a Ruby place We
00:07:11.039 wanted to be able to use all this Ruby
00:07:13.360 code that we had in our monolith and we
00:07:15.759 wanted to keep doing Ruby
00:07:18.160 And here we arrived to let's call it the
00:07:21.520 strange place the red room and in there
00:07:25.120 were like scary
00:07:28.039 creatures First one and if you've been
00:07:30.000 doing Ruby for a while you will you may
00:07:31.680 remember those times was memory
00:07:34.759 management uh in those times you know
00:07:37.039 memory leaks were uh were very
00:07:38.560 problematic and since then I would I
00:07:41.039 would like to say that it improved a lot
00:07:43.280 Um I remember reading articles uh from
00:07:46.479 before I started writing Ruby people
00:07:49.360 saying that you need to restart your
00:07:50.800 Rails app every few hours or something
00:07:52.240 like this That's obviously not the case
00:07:55.000 anymore But the reality is that
00:07:57.520 particularly in computer science uh you
00:07:59.360 never get a second chance of of making a
00:08:01.440 first
00:08:02.360 impression But anyway I didn't want to
00:08:04.720 talk about memory management so much I
00:08:07.039 wanted to focus on a more scary monster
00:08:09.599 that was waiting for us out there when
00:08:12.000 we wanted to build this
00:08:13.800 service The
00:08:16.360 GBL I I is a bit tongue and cheek Later
00:08:19.520 we will talk a bit more about this But
00:08:21.680 the thing is that the GBL as you know it
00:08:23.440 ensures that uh only one thread executes
00:08:25.440 Robi code at a
00:08:27.160 time And well before I was talking to
00:08:30.319 you about AI So for to introduce to you
00:08:33.200 the the GBL I will show you a video
00:08:35.760 generated by AI uh that basically says
00:08:39.039 create video of a beginner developer new
00:08:41.440 to Ruby meeting the GBL for the first
00:08:43.839 time
00:09:01.760 Okay so the monster here was not so much
00:09:03.519 the GVL but multi-threading You know
00:09:05.760 doing multi-threading in Ruby a multi-
00:09:07.600 threaded application could be particular
00:09:10.160 of times quite
00:09:11.800 complicated Again I you shouldn't see
00:09:14.080 the GBL as an enemy that makes Ruby slow
00:09:16.640 Actually for single thread applications
00:09:18.320 it probably makes it faster And actually
00:09:20.240 if you want to know more about it I
00:09:21.760 would recommend you to read the his blog
00:09:23.680 like Jam Busher's blog You will learn
00:09:26.080 way more than in this than this
00:09:27.680 presentation
00:09:29.680 Moreover uh nowadays on it's been the
00:09:32.160 case for more than 15 years the GBL is
00:09:34.240 actually released for certain operations
00:09:36.399 So for example when you are doing a
00:09:38.240 operation system calls sleep operations
00:09:40.800 or or if you call well this method uh
00:09:43.839 you can uh unlock undo the the lock The
00:09:48.560 thing is that and this is something that
00:09:50.160 is very important I believe to
00:09:51.600 understand the Ruby the discussion about
00:09:53.519 about Ruby multi- threading is that it
00:09:55.600 the case wasn't it wasn't it wasn't
00:09:57.920 always like this uh in the late 2000s
00:10:00.800 actually until December 2007 uh Ruby had
00:10:04.080 green
00:10:05.160 threads that meant that Ruby didn't have
00:10:07.519 real threats so uh you know IO calls
00:10:09.920 well were not unlocking the the
00:10:12.399 processing um this wasn't the case for
00:10:15.279 other dynamic languages like Python
00:10:17.839 never had something like this and I
00:10:20.720 believe that that's why that's the
00:10:21.839 origin of why you know the the scars in
00:10:24.560 place places like hacker news or Twitter
00:10:26.560 about Ruby it's sometimes so negative
00:10:28.720 about
00:10:30.040 performance still at the time there was
00:10:32.320 a very popular solution I remember it
00:10:35.279 because uh the year I took this photo uh
00:10:38.160 it was in the Barcel Ruby conference
00:10:39.920 actually uh there were multiple talks
00:10:42.560 about about this as a as a possible
00:10:45.680 future for for Ruby
00:10:51.760 This was a solution for us J Ruby
00:10:54.640 offered to us real
00:10:57.480 threats and not only it was not only an
00:11:00.079 issue with uh with speed with or with
00:11:02.079 concurrency in this case It also offered
00:11:04.000 to us access to Java libraries which at
00:11:06.560 that point in time was great because
00:11:09.120 Ruby didn't have that many you know
00:11:11.200 external libraries things that we needed
00:11:13.920 language detectors as I told you before
00:11:16.320 we we were going to run textual search
00:11:18.720 and for doing that we need to identify
00:11:20.560 the language in which the text was
00:11:22.399 written and also it offered to us the
00:11:25.360 elastic search Java client those were
00:11:27.279 the early times of elastic search and
00:11:29.360 and we the the Java client of elastic
00:11:31.600 search had b processor that we needed to
00:11:33.839 run this on a scale most importantly
00:11:36.959 well uh Jub was and still is
00:11:39.519 significantly faster this is a benchmark
00:11:41.600 I found
00:11:43.320 2012 So all in all we decided that we
00:11:47.120 would do it and we would write our
00:11:49.079 application using J Ruby We decide to
00:11:52.320 enter the this mysterious
00:12:08.360 room and it as well Again this the
00:12:12.000 origin of this talk uh is actually
00:12:14.959 because I was watching a TV show um and
00:12:18.000 it's related with the amount of years
00:12:19.440 that we that we've been maintaining this
00:12:21.839 it's been running at Zenesk scaling as
00:12:24.560 much as Zenesk needed for 15 years Uh
00:12:27.920 the origin of the talk is a bit because
00:12:29.839 I was watching this TV show Maybe
00:12:31.600 someone realized which TV show was this
00:12:34.399 video is about the AI video was
00:12:36.399 obviously not generated by AI This is
00:12:38.639 actually from Twin Peaks and actually in
00:12:41.279 the last episode of the original run it
00:12:42.880 was said that I will see you again in 25
00:12:46.040 years Again the service has been running
00:12:48.560 for us with for 15 years with not that
00:12:51.920 many changes which I find pretty pretty
00:12:54.399 great news Still we've learned we've
00:12:58.000 learned some some lessons of running a
00:13:01.040 system in production in using J Ruby at
00:13:03.600 the scale that this this thing runs What
00:13:06.560 scale well the title of of this
00:13:08.959 presentation says it Uh in the peaks
00:13:11.279 we've been running 50,000 updates in 1
00:13:13.200 second Most typical um daily max is
00:13:16.320 around 15,000 and almost continuously
00:13:18.880 we're running 7 7K per second It's a
00:13:21.279 pretty common average Uh moreover in
00:13:24.720 your typical day this service uh
00:13:26.560 executes around 600 million database
00:13:28.639 queries across all all all
00:13:32.040 charts Still again we had our own share
00:13:34.800 of issues Which ones well multiple Uh
00:13:39.760 one that I like to say it's a kind of a
00:13:41.760 a joke at this point for us is that for
00:13:43.920 a while we tried to unify coding styles
00:13:46.880 Um we had to write a little bit of Java
00:13:49.839 code because we we needed it Uh some
00:13:52.560 sometimes we got even Java developers
00:13:54.320 working on it and we end up writing
00:13:56.720 something I
00:13:58.600 call or something like this I actually I
00:14:01.600 actually got this because I asked uh
00:14:03.440 Chachip to draw the you know the Ruby
00:14:06.240 logo with a with the Java style But
00:14:09.440 anyway that wasn't that wasn't great Uh
00:14:11.839 it's actually hard to the eyes of a
00:14:13.920 Rubist to see uh you know Java style
00:14:17.800 Ruby It's it's kind of strange Um like
00:14:22.880 this is just an example of something
00:14:25.360 super common where you get Java style
00:14:27.440 Ruby like well this is an abstract class
00:14:29.680 So you get a lot of abstract classes a
00:14:32.079 lot of
00:14:32.920 interfaces I mean I don't know how you
00:14:34.880 feel but I feel more like
00:14:37.000 this Um another issue that you have
00:14:39.760 commonly is uh library
00:14:42.680 diverence So for example you have
00:14:44.880 certain libraries that you can only
00:14:46.240 access in J Ruby and the typically this
00:14:50.399 is are Java libraries but also you have
00:14:53.279 the other situation libraries that only
00:14:55.600 exist in in Ruby in C Ruby let's call it
00:14:59.279 I would like to tell you what I call the
00:15:01.120 sik story of of our service Does anybody
00:15:04.959 know what is sik syk
00:15:08.399 well so basically is a is a YAML parser
00:15:11.360 that it came originally well um it was
00:15:14.480 the original YAML parser for Ruby Uh it
00:15:17.760 was part of standard library for quite a
00:15:19.360 lot of years It hasn't been that for 10
00:15:23.079 15 and now it's been replaced by
00:15:25.440 something called psych and there are a
00:15:27.760 few differences like 99% of the time
00:15:30.399 those two channel parsers are going to
00:15:32.800 have the same behavior but that 1% is
00:15:35.440 incredibly important when you are again
00:15:37.600 processing 50,000 records per second and
00:15:40.480 more uh more importantly those records a
00:15:43.279 lot of time comes from customers which
00:15:45.519 can write whatever it's not something
00:15:46.880 you can control um eventually to be able
00:15:50.240 to process this I actually try to ask uh
00:15:52.720 support for the the new YAML the old uh
00:15:56.160 YAML parser So right now this old YAML
00:15:58.880 parser only exists in C Ruby You cannot
00:16:01.199 use it in in J
00:16:02.759 Ruby This was a problem that so so bad
00:16:05.440 that eventually I had to build a
00:16:08.040 transformer in between these two YML
00:16:10.160 parsers So uh content that have been
00:16:12.480 serialized using sik could be read by
00:16:15.120 sik uh by psych uh parsers Another thing
00:16:20.000 is that we found ourselves attached to
00:16:22.639 Java library Sometimes we say hey maybe
00:16:24.480 we should try to go you know to write
00:16:27.480 pure Ruby J Ruby like let's let's forget
00:16:30.639 about all all this Java all the Java
00:16:33.040 code but uh the problem is that
00:16:35.680 migrating to a new library may have
00:16:38.480 different behavior for example the
00:16:40.079 language detector um most probably the
00:16:43.120 majority of the time we have the same
00:16:44.480 behavior but we may break beh uh certain
00:16:49.680 use cases so we don't want
00:16:51.720 that moving to a Ruby equivalence was
00:16:54.160 was hard But maybe the most important
00:16:56.959 issue here are thread safety issues We
00:17:00.000 had to work all the time around the
00:17:01.839 threat the possible threat and safety of
00:17:03.440 some libraries This is less less the
00:17:06.480 case right now but particularly in the
00:17:08.720 2010s uh we we found that some Ruby
00:17:11.360 libraries were threat and safe and
00:17:12.720 sometimes we were not even conscious of
00:17:14.480 that they
00:17:15.880 were So we had to learn to write thread
00:17:19.199 safe Ruby I think that historically
00:17:21.839 again that's not the case anymore
00:17:23.919 concurrency has not been a first class
00:17:25.679 citizen in
00:17:27.400 Ruby Okay And with this we get to the to
00:17:30.480 the to the to the final part like what's
00:17:33.440 in the future what we are building Uh
00:17:36.160 well the thing is that it's possible
00:17:37.440 that we go back to the models I think
00:17:39.280 that a lot of the the ideas that were
00:17:42.520 behind let's break the let's break this
00:17:45.600 uh behavior and build a new application
00:17:48.240 I think that's not that's not valid
00:17:50.120 anymore Why well there were multiple
00:17:53.039 reasons to buy to build microservices in
00:17:54.880 in the
00:17:55.880 other For example people wanted mobile
00:17:59.039 and and team autonomy Like people says
00:18:01.520 "Hey a group of developers will move
00:18:03.760 faster if you know they each each one of
00:18:06.000 them has their own application or maybe
00:18:08.160 not each developer but each team." Well
00:18:11.120 I kind of disagree If you want to know
00:18:13.120 more I really recommend you this talk
00:18:14.960 from Rails World last year Uh it's
00:18:17.120 called the myth of of the molar morith
00:18:19.520 is by by a um she's commenting on on
00:18:24.240 modity on molar mories using uh for
00:18:26.840 example um uh pack work
00:18:30.640 uh but the point stands her point stands
00:18:33.600 her main point for me was that you
00:18:35.280 cannot engineer yourself out of a large
00:18:37.400 organization and this problem will will
00:18:40.160 stay So in in this regard uh you know
00:18:43.039 breaking new applications out of a big
00:18:44.640 one doesn't help I I do find value in
00:18:48.320 things like pork into establishing some
00:18:50.640 kind of live mobile ID and if you have a
00:18:53.280 large application but you know it's a I
00:18:56.720 still believe that she's right like no
00:18:59.440 going extreme in the mobile ID path is
00:19:01.440 probably not not going to give you you
00:19:03.760 know a good
00:19:05.160 reward Another issue why uh
00:19:08.720 organizations wanted to create new new
00:19:10.960 applications like breaking these smalls
00:19:13.760 was uh having flexibility in tech
00:19:15.960 choices and unfortunately those times
00:19:18.240 meant that some Ruby organizations or
00:19:19.840 companies that have been started using
00:19:22.400 building in Ruby that meant that they
00:19:24.160 didn't want to use Ruby anymore but what
00:19:26.799 we found is that other you know other
00:19:29.360 things that became new in the in the
00:19:30.960 field during the last decade help a lot
00:19:33.039 with scalability For example the cloud
00:19:35.600 the the combo of cloud and
00:19:37.559 Kubernetes The following is a is
00:19:39.919 actually a quote from from one of our
00:19:42.559 group tech leads at Zenesk like with
00:19:44.799 Kubernetes in the cloud we don't need to
00:19:46.640 care that much about multi-
00:19:48.200 threading and actually would say that
00:19:50.240 maybe you don't you don't even need the
00:19:51.960 cloud Uh the the one of the improvements
00:19:55.120 of the last five years I would say is a
00:19:57.200 lot of improvement on tooling
00:19:59.520 uh when when some companies moved to the
00:20:02.000 to the cloud uh the perception was that
00:20:04.880 you know things like AWS offer you a set
00:20:07.520 of tools that maybe you couldn't have in
00:20:09.200 house that's less less of the case uh
00:20:12.160 actually if you if you're interested in
00:20:13.600 this DHH is blogging quite a lot about
00:20:16.080 their their move of 37 signals out of
00:20:18.559 out of the cloud and looks like they are
00:20:21.120 going they are going very well in this
00:20:23.559 regard finally and again this is another
00:20:26.240 solution that comes from the
00:20:27.280 architectural right uh is the rise the
00:20:30.000 rise of event sourcing I mean
00:20:32.000 technologies like like
00:20:33.559 Kafka why is this useful for services
00:20:36.640 well basically with this you can kind of
00:20:39.039 can replace a a pull paradigm with a
00:20:41.520 push which reduces a lot the load on on
00:20:44.559 data sources like the like the
00:20:46.679 database it also allows uh a very simple
00:20:50.240 way to run patch processing
00:20:54.320 um moreover if you are going to back to
00:20:56.159 hey I would like to focus everything to
00:20:58.080 center everything around a Ruby
00:20:59.880 monolith You can have those C consumers
00:21:02.240 in the Ruby monolith
00:21:03.720 itself I see about that I would like to
00:21:06.159 commend on a few uh libraries that have
00:21:08.799 been appearing in the last years One
00:21:11.120 that one for example uh that appeared in
00:21:13.400 the some time ago is Maxwell with
00:21:16.240 Maxwell if you are interested in event
00:21:17.760 sourcing and particularly event sourcing
00:21:19.919 data used to be in in the database it's
00:21:22.880 great What Maxwell does is it reads from
00:21:25.600 the the bin lock of a database and
00:21:27.840 generates one event for each uh each uh
00:21:31.240 transaction with it You can have a you
00:21:33.440 know a onetoone uh connection between uh
00:21:36.240 you know your Kafka topic and your
00:21:38.919 database Yeah Finally uh I would say
00:21:42.080 that C Ruby is faster and way more
00:21:44.159 concurrency friendly than than it used
00:21:45.919 to be
00:21:48.720 Um and there is m you can see that in
00:21:51.200 the amount of you know new things that
00:21:53.520 are happening in the Ruby community that
00:21:55.440 that go in that direction For example
00:21:58.159 one is
00:21:59.480 Yet like at Zendes we implemented Yet
00:22:02.480 and we were able to reduce CPU usage by
00:22:05.880 20% Another is another is uh concurrency
00:22:10.320 concurrent Ruby that has been around for
00:22:11.919 a while already And finally one thing
00:22:14.559 that I I am very excited about and I'm
00:22:16.799 maturely rooting for is
00:22:19.000 Ractors This is something that maybe we
00:22:21.120 are not using still in production but
00:22:22.720 it's something that we're always you
00:22:24.400 know checking how how is it going I
00:22:26.640 think that this could be the the future
00:22:27.919 of you know high performance Ruby in the
00:22:30.280 future But even in libraries that
00:22:32.720 probably doesn't seem to be directly
00:22:34.559 related with multi-threading and
00:22:35.919 performance you see that there is a huge
00:22:37.600 effort Uh one thing I I love is uh here
00:22:41.960 uh is here uh Karafka Karafka is the
00:22:45.440 most the most popular uh uh library for
00:22:49.280 uh uh um for uh processing Kafka Kafka
00:22:52.640 messages And there is a if you check in
00:22:55.000 documentation there is a section only
00:22:57.679 about concurrency and multi- threading
00:22:59.600 This is I for me it shows how how much
00:23:02.159 uh seriously we are taking performance
00:23:04.400 in the Ruby community Okay Yeah We're
00:23:07.919 getting to the
00:23:09.000 end but I would like to give you to
00:23:11.440 leave you with a few a few ideas I would
00:23:13.760 say that Cub limitation have been
00:23:15.919 grossly exaggerated Like uh again I go
00:23:18.480 back to this idea of the public
00:23:19.600 discourse about Ruby part the public
00:23:22.159 discourse about Ruby outside of the Ruby
00:23:24.840 community more than grossly exaggerated
00:23:27.039 They feel that sometimes it's a debate
00:23:28.559 that it stayed in the stone age or maybe
00:23:31.039 you you feel like you are debating with
00:23:32.960 people that are fighting against Ruby
00:23:35.280 1.7 and uh Rails 2 Uh it's problems that
00:23:39.520 existed a certain point in time but
00:23:41.440 hasn't haven't been the reality of Ruby
00:23:43.919 for a long long time
00:23:46.960 M moreover as I told you before I I
00:23:48.799 focus on scalability issues and I will
00:23:50.799 tell you that vast majority of
00:23:52.000 scalability issues that are openly
00:23:54.080 discussed again on the internet are
00:23:56.320 absolutely imaginary vast majority of
00:23:58.720 businesses are not working uh are not
00:24:00.799 operating on the scale that uh Twitter
00:24:03.559 operated so it's problems that literally
00:24:06.960 didn't happen
00:24:08.280 never but here the the positive note
00:24:11.039 that I will like to leave you is that
00:24:13.279 the Ruby community has worked a lot on
00:24:15.039 these issues for years I was just having
00:24:17.039 lunch uh you know with some work
00:24:19.760 colleagues a few one hour ago and we
00:24:22.159 spent some time debating about you know
00:24:23.679 changes on on the on on on Ruby how
00:24:26.400 we're becoming well more performant
00:24:28.480 faster how we're optimizing that's
00:24:31.240 amazing but finally in the end uh my
00:24:35.360 what I would like to leave you is that
00:24:36.960 you should choose whatever tool is most
00:24:38.960 important for is better for the project
00:24:41.440 in some cases yes it may be the one that
00:24:43.840 works faster the one that is it can use
00:24:45.760 the resources better But sometimes it
00:24:48.400 can be something different Sometimes you
00:24:50.240 can choose whatever is best for the team
00:24:53.520 because again engineering productivity
00:24:55.600 is very important to to generate great
00:24:58.760 software and in this regard I think that
00:25:01.279 Ruby has a very significant advantage I
00:25:04.480 would like to leave you with this quote
00:25:05.919 by Matt Ruby's designed to make
00:25:09.279 programmers
00:25:10.840 happy Thank you
Explore all talks recorded at RubyKaigi 2025
+66