00:00:19.760
present, and future of background jobs. Uh, according to Mike. Oh my god, we're
00:00:25.920
in order. Mike, Rosa, Magic, and I'm Ben. Um, so just agenda wise, um, we
00:00:34.320
want to try to do about 30 minutes of canned questions and then, uh, really
00:00:40.160
open it up to the audience for the last half of it. Um, so I'm just going to jump into questions as part of our
00:00:46.879
introduction. So, uh, if we could introduce ourselves and the job systems
00:00:52.160
that we work on and what do you think makes them unique or different from the
00:00:57.600
rest of us? Um, I'll start with you, Mike. Thank you, Ben. I'm Mike Pam. I am the
00:01:04.000
author and maintainer of Sidekick. Uh, I also have a business around Sidekick, selling Sidekick Pro and Sidekick
00:01:10.400
Enterprise. Um, what makes my job system unique? I think uh I'm the only one up
00:01:15.439
here that depends on Reddus. So I use Reddus. I like Reddus. Uh it's a extremely useful data store and uh it's
00:01:23.280
really good at building a job system uh or powering a job system.
00:01:30.799
Okay. Hi everyone. I'm Rosa. Uh I'm the author and maintainer of Solid Q. Um I
00:01:38.479
think what does what makes Solid Q different right now? Um before my go-to
00:01:45.280
answer was that it uses a database but that is no longer unique here. So I will
00:01:50.399
say it's the current default in Rails and it's it can be used with uh
00:01:58.159
different databases like currently we support my SQL possess and SQL light. So
00:02:04.079
yeah I have my own. Hi guys. Um, I'm Magic.
00:02:09.599
I'm the creator of Karafka, which is the Kafka Ruby framework, and the newly appointed maintainer of Shyukan, which
00:02:15.760
is the AWS SQS1. So, I'm representing myself uh and a bit of a representing
00:02:21.440
Pablo, the creator of uh Shyukan. What makes my uh job systems unique? Uh yeah,
00:02:30.480
first of all I think that in terms of uniqueness, Karafka is more than a job
00:02:36.959
execution system. Uh so that's that's definitely something that makes it unique because it provides like
00:02:43.200
capabilities beyond uh worker or job execution and uh what makes Shyukan
00:02:50.400
unique? I think one of the things that make it unique is that the fact is that it is a really old fork of Sidekick. Uh
00:02:57.120
really old. Uh Mike fixed all of the issues that I now have to fix as a newly appointed maintainer. Uh
00:03:03.440
you're maintaining my code. Yeah, in a way. In a way. Yeah. Uh yeah, but what I also think is really unique
00:03:09.760
is that it was almost abandoned. Uh I took it over and if there are any users
00:03:15.920
of Shyukan, please ping me because I need to re rebuild the community from the ground.
00:03:23.040
Um hey everybody. Uh Ben Sheldon. I'm the author of Good Job. Good job is a
00:03:28.959
multi-threaded Postgress based uh active job system. Um I'm like wow I think
00:03:34.159
there's less unique stuff than if I had given this uh two years ago. Um but it's Postgress specific. So it uses some
00:03:41.440
Postgress features. Uh it uses listen notify I use advisory locks um for them.
00:03:48.319
And yeah, uh I'm going to move to my next question, which was uh if you try
00:03:53.840
to remember back, um for each of you, uh was there some sort of event or
00:03:58.959
motivation that caused you to like actually build these things in the first place?
00:04:06.400
Uh yeah, there was uh I was working as a consultant uh with a client that had a
00:04:11.519
pretty large installation of rescue and rescue is singlethreaded. So you
00:04:18.160
fork a process for every job and they were trying to run thousands of jobs. So they had thousands of multi- gigabyte
00:04:25.199
processes and it was just taking an insane amount of RAM. Uh and that's where I said maybe multi-threading would
00:04:33.120
really help here. Uh since you can share the memory amongst jobs and that's where
00:04:38.320
the um uh the the core of Sidekick, the idea of Sidekick came from. Um,
00:04:46.800
was that the answer to the question or was there more to the question? I think that was the question. All right.
00:04:54.000
Yeah, my answer is uh not as nice. I suppose in my case, what motivated
00:05:00.160
building solid Q was that David has this had this idea for it and he paid me to
00:05:06.080
do it. I work for 37 signals, so he's my boss and so this was a project I got. Um
00:05:13.120
but actually what motivated it in the first place was that we use res we use
00:05:18.240
rescue for most of our apps. Rescue is is really I I like it a lot personally but uh as as Mike said uh we had
00:05:25.600
problems with the multi-threading stuff. We have to run multiple processes. We have a lot of uh custom gems that we
00:05:33.199
built to overcome some of the son of rescues limitations and it had become
00:05:39.360
pretty complicated. So when we started a new app and uh we saw that we had been
00:05:46.320
um porting like bringing the the seven gems or so that we had and a lot of
00:05:51.840
customs that for rescue over and over to every app they David wanted something
00:05:57.039
that just work out of the box in a much simpler way and we decided to start
00:06:03.840
something from scratch now with all with everything that we've learned from running many running jobs in many
00:06:11.360
different apps. Yeah, I have a really simple answer to
00:06:16.960
this question. Hyperdriven development, you know, like my friend came to me, he's like, "Hey, there's this new thing
00:06:23.199
called Kafka. Uh, do you want to use it?" And obviously I didn't want to do the business logic of the company I
00:06:28.479
worked for. Uh, so I I needed to find a reason. Uh, so I built Karafka exactly
00:06:34.319
for that reason. Um, there was like a really low-level driver. Uh but it it's
00:06:40.000
development experience was bad and instead of trying to convince people that we could actually use it, I just
00:06:45.440
wanted to wrap it around. Uh and for sure you can uh I think I can speak on
00:06:52.000
behalf of Pablo. You didn't want to accept a plug-in for AWS SQS, so he just
00:06:58.000
forked that uh pretty much. So it's all on you. Yeah.
00:07:05.280
Uh with good job. I think the I really wanted something that I could run on
00:07:11.039
Heroku. This was back when there were free dinos that would like be able to run um in a web process. And there was
00:07:17.840
another job q system called qe that I really liked. And that's sort of like
00:07:23.599
the maintainership of that um was sort of absent. And I I was like for a hot
00:07:31.440
minute I was like maybe I could be the maintainer of this. And I looked at the code and I was like this is way too complex for me. Um, and I sort of had an
00:07:39.440
experience at work where I was doing stuff with advisory locks and I was doing stuff with concurrent Ruby and I was like, "Oh, it would be really,
00:07:44.960
really, really simple to just like put these things together um, and have
00:07:50.000
something that I could just like run in the Puma web process on a cheapo Heroku
00:07:55.680
dyno." Uh, and that was sort of how it started. Uh, so I'm going to ask my next
00:08:01.759
question, which was, um, so that was how things started. uh now where we are
00:08:07.440
today um over your experience of like developing and supporting uh your system
00:08:12.879
or systems like what has most surprised you um or what has sort of changed from
00:08:19.120
your original vision or motivation if anything? I yeah I'm struggling with this question
00:08:24.960
a little bit. Um I'm a little bit of an odd bird here because I have a business built around a
00:08:31.199
Ruby gem and there's not a lot of people in the world that have that. Uh, there's one more here.
00:08:36.880
Fair enough. Fair enough. 50% of this. Oh, yeah. There's there's a couple of people selling pro versions. Um,
00:08:43.279
but uh I'm I'm just happy that the community by and large accepted that. Um, because it really makes me able to
00:08:50.480
m, you know, I've been maintaining Sidekick for 12 years now. There's no way I'd be able to do that if I was if I
00:08:55.519
was working uh for free uh to put in all the hours and all the polish that I've put into it. So I I'm happy that the
00:09:03.040
community by and large accepted um my commercial offerings.
00:09:10.640
Um yeah, for me the the the biggest so the biggest surprise or the biggest
00:09:19.200
change from the initial vision for sure was to um renounce to to give up the
00:09:27.200
transactional integrity when queuing jobs like this is something I remember I've discussed in issue or pull request
00:09:34.000
or whatever with them because I lied a lot uh like using the database to store
00:09:41.120
jobs. One of the things I liked the most in the beginning uh was that you could
00:09:46.399
use like if you store your jobs with your data in the same database and you could use transactions to ensure like
00:09:52.560
for example if the job fails to be encued you roll back the the other data
00:09:58.240
or things like we we had some problems uh in the past when we assumed that the
00:10:04.560
job was going to be encued no matter what and then we made assumptions on that but sometimes you commit something
00:10:11.519
to the database you inque a job and maybe ready is down when we were back using rescue and so you need to manually
00:10:18.000
roll back the other data this kind of things so I was since I have deal deal with incidents around that in the past I
00:10:26.000
was very excited about that so in the beginning I built the whole thing around this you know assuming I will have this
00:10:33.360
but then uh rails core were very much against this this this this way this
00:10:41.360
this uh feature. Um I think it was because they some people in their score
00:10:47.120
used to have h used to use delayed job and they were inadvertently relying on
00:10:54.399
this transactional integrity for some things and when they mig migrated to a
00:10:59.519
different system they they had a lot of bugs there. So they were traumatized by that. I understand I've been traumatized
00:11:05.519
by job problems as well. So they had this very strong opinion on not having
00:11:11.760
this by default. So I have to change that and in the end I have to change it anyway because we have to move to a different database. But the thing is
00:11:18.160
that it was completely uh 180 um turn
00:11:24.720
from what I wanted to do in the beginning and yeah and pretty much embraced the not relying on
00:11:31.600
transactional integrity. Even even now with added job by default I think jobs
00:11:36.720
are incued after commit uh so after all transactions are committed even even if you encue them in the middle and so yeah
00:11:43.600
that'll be it okay uh for me uh few things actually
00:11:50.000
there's like part of this question that wasn't read technically socially or anything right so socially I'm quite
00:11:56.959
quite optimistic about how people accepted the pro version of my software
00:12:03.680
uh this made all the difference for me and how much I time I can invest and how
00:12:09.360
good of a quality of software I can provide when I have resources to do that. Uh what surprised me also and it
00:12:16.560
still keeps surprising me is that a lot of majority of Rails or Ruby community still don't understand how good
00:12:22.079
ecosystem we have for Kafka actually how way ahead we are in terms of like developer experience and many other
00:12:28.160
things compared to any other language there is uh and for sure you can uh I
00:12:35.279
don't have a vision for it yet a huge vision uh what surprises me is that how
00:12:41.519
old-fashioned it is and how much work Mike did uh since the fork. Uh and the
00:12:47.519
fact that it doesn't have a UI. This is a big problem for uh working with it and
00:12:53.920
operating it is like yeah unacceptable. I'm going to fix that.
00:13:00.160
Uh with good job. I think the thing that surprised me the most was how much else
00:13:08.160
people wanted from it. that like when I originally like I said I was like oh
00:13:13.360
it's just concurrent Ruby and a Postgress query and it was like I think the first version was like 500 lines of
00:13:18.880
code and it was just background jobs and then like you know I'm glad people started using it but like almost
00:13:24.959
immediately people were like oh we also need like cron like scheduling and then people wanted batches people wanted uh
00:13:33.760
the concurrency controls and I was very sort of just like no it just does
00:13:40.480
background jobs. Uh, and sort of like held the line. Uh, but I was sort of like I guess I was convinced. And I
00:13:47.440
think the the piece was just like, okay. Um, like I say, like my target was
00:13:53.200
really people that wanted to just run a cheap website and do it well and
00:13:58.800
realizing just like, oh, actually, if you want to not do it perfectly, but just like do it well, do a good job of
00:14:05.199
it. um that there was yeah there was other uh other components that I did
00:14:12.079
sort of come around to realize and like no you actually do sort of like definitely chronike jobs um and now I
00:14:18.560
think good job is like 20,000 lines of code and some parts of it I am much less happy about the implementation with than
00:14:25.360
that that pristine 500 lines of code. Um, but I guess like just sort of like
00:14:30.720
once you start on that path, um, I guess maybe Mike, this is something you've been doing sidekick forever that it's
00:14:36.639
like people just there are there is more that you can always be doing to make it better. And I was, yeah, surprised by
00:14:43.440
that. One of the hardest parts of maintaining a large system like you say, Ben, is
00:14:49.839
having a vision and sticking to that vision. And if people do ask for functionality like you say,
00:14:58.079
melding it into something that would fit into your vision or alternatively saying
00:15:03.199
no, right? Sometimes it's really hard to say no but you have to. Um I have a a
00:15:09.519
third position which is I can say you've got to pay me for it, right? So that's the other thing is is if it's if it's hard the batches API the
00:15:17.760
batches API in Sidekick Pro was really hard to develop. It took me probably five years before I got that API
00:15:24.079
stabilized. Um, and so I'm I'm I have no um qualms about making that a commercial
00:15:31.360
API because it was just really hard to build. You can always tell them to pay and then
00:15:36.399
then say no. You know that that's true too. Wonders of free markets.
00:15:43.519
All right, this is the the spicy question. Um, what do you think uh
00:15:49.120
people get most wrong or misunderstand uh or require the most handholding or you just wish that people like did
00:15:55.199
something differently? Uh, yeah, I think the one of the biggest
00:16:03.519
blockers for sidekick usage is it's just reliance on Reddus. I think a lot of people uh are comfortable, they
00:16:10.320
understand that they're going to need a database. Um but then they ask themselves why do I want to run Reddus
00:16:15.600
too? I I think it in itself has a lot of value as a data structure server. Uh
00:16:21.199
applications can really build a lot of interesting functionality on top of the building blocks that uh Reddus gives
00:16:27.360
you. Um you know I built a job server on top of the the building blocks that Reddus gives me. Um but I I think people
00:16:35.440
are a little hesitant to try Reddus and really it's incredibly stable. It's
00:16:41.199
incredibly fast. I I love that software and a lot of people do too. But uh I I
00:16:46.639
would just encourage people if they've never tried Reddus before to to give it a give it a try and and maybe read
00:16:52.000
through the documentation and and see uh if it maybe gives them ideas for interesting or novel ways to solve
00:16:58.160
problems they're having in their app. Yeah. And a plus one to that. I love
00:17:04.799
Reddis. It's so nice. It's such a great piece of software. Um, for me, I think I
00:17:10.959
that was also a big surprise for me. The the thing that people that takes the most work and and people seem to
00:17:19.439
struggle the most is that so when with solid Q we use uh by default it uses
00:17:26.799
different databases a different database from your app. So we uh you need to set
00:17:31.919
up to to set a different database um for rails and rails support this uh by by
00:17:38.720
like it's built in in rails and and then solid q doesn't have a m doesn't have
00:17:44.880
migrations. It just use a schema file to load the database because it's just
00:17:50.720
starting the schema file. Um but for some reason first we uncover a few bugs
00:17:58.160
in the in rails multi database handling because it seems that before solid Q and
00:18:04.080
solid okay that's yes solid Q and solid cache started setting multiple databases
00:18:11.440
people weren't using this very much I think because there was a few bugs that were uncovered after we we um released
00:18:19.280
the first version with this and people starting using it. That was one thing. But even after fixing those, um, people
00:18:26.799
really want to run migrations for it. And you don't need to run migrations because there are no migrations and you
00:18:33.280
just need to to load the the the database, the schema file with Rails. It's just a command. But I still get
00:18:39.919
questions almost every week about why the migrations are not running. And I I
00:18:45.200
don't know. It's it's true that it's a little bit odd. Um, but I I I I don't
00:18:52.960
know. I thought it was clear like in the rhythm and everything. But yeah, it's it's the thing that people struggle the
00:18:58.720
most just setting it up and getting the database ready.
00:19:04.960
Maybe that's because you Rails has engines, right? And they have migrations. So that I I think the
00:19:10.559
natural expectation would be you plug something in and it operates on a database. So you should have a
00:19:16.480
migration, right? just add one and you you'll not have any more questions. Uh now you know
00:19:24.240
now it just become a matter of because it works with the schema is the same. You just need to prepare the database
00:19:30.400
run DB prepare load the schema as you will do with a new database and I agree like if I could just add a migration
00:19:36.400
yeah just one just one but now I'm now anyone you know no now I will die on this hill like now
00:19:42.240
the schema works now I I'm just it really works it works like you can just
00:19:47.679
use it I'm not going to create a migration because there is nothing to migrate there is just a new database you
00:19:53.679
can just load the schema and it works it works
00:19:58.799
Okay. Uh let's just keep it. Uh for me uh one of the bigger
00:20:05.760
misconceptions uh that I'm trying to fight with is that people are using Kafka as a message Q as a job queue only
00:20:13.360
which is not its primary usage. Yes, I build an active job support for it. Uh because people do use it but it
00:20:19.120
shouldn't be the primary case. So if you're thinking about uh using it just to run jobs then don't pretty much uh I
00:20:27.039
spend a lot of time and the second thing is uh no one reads the documentation you know I I have beautiful dogs uh no one
00:20:35.039
reads them uh which makes me really really sad and uh this requires a lot of
00:20:41.600
handholding to teach people that uh I am writing docs for a reason uh and then
00:20:47.120
reason is basically the users uh for sure you can. Uh I don't know yet. Uh I
00:20:54.559
I think that what people got wrong with it is that the expectation that the
00:20:59.840
maintainers will do everything for free. They will never burn out and they will be happy just by you know getting stars
00:21:05.280
and giving those stars to their children to eat. Uh so yeah I I think people get
00:21:11.760
that wrong. I think on my side um this I wrote this
00:21:18.320
question for me to have a soap box um because I have a very strong opinion which is uh you should name your job
00:21:25.440
cues based on latency not on uh like what they do. So don't
00:21:32.159
have a mailers cue. Have a 30 second latency queue uh for the emails that
00:21:39.360
you're like really want to go out quick like a reset password email. Uh, and then have a like 10 minute latency or 1
00:21:47.200
hour latency or whenever latency. That's what I call mine for like your bulk
00:21:52.640
email like mailer. Uh, if you're emailing, it's like bulk where you don't actually care when it goes out as long as it just goes out. Um, but I see this
00:22:02.159
all the time of people just uh usually people this is uh was it where it's like
00:22:07.919
somebody asks one question but it's really because of a misunderstanding with something else where they're just like how do I do quality of service with
00:22:15.200
my mailers queue? Um and it's like well you don't you just have two separate
00:22:20.240
cues. Um, so please uh do latencybased
00:22:25.679
um organizing of your Q names. Please please do that. And shout out to Gusto. I believe they
00:22:32.159
popularized that that idiom. So thanks.
00:22:37.520
Uh all right. Um I'm skipping the question on the fold. Uh next one we've
00:22:44.480
got uh so if you look ahead 10 years, we're into the future part. Uh if you look 10 years into the future and
00:22:51.360
assuming that Rails and Ruby are still doing good, which we are assuming here, uh how do you imagine your background
00:22:58.880
job system might be different? 10 years sounds like a long time. Um
00:23:06.080
Sidekick has reached a a pretty good point of stability. Uh there's not a lot of new features going in. I did just
00:23:12.240
release an 8.0 know, with a couple new features, but um by and large, I'm not not seeing a lot of new features. It's
00:23:19.039
really dependent on the community. If you're using Sidekick and you want want something uh open an issue that, you
00:23:25.440
know, you're the driver really of the road map. Um but the the main, you know,
00:23:30.880
thing that sticks in my cross still is the gill. The gill really impacts the the um effectiveness of multi-threading
00:23:38.480
in Ruby. And so, uh, I've I've been trying to work with Ractors for a couple
00:23:44.720
years now, and they're they're slowly improving, but it's it's a long slog. Uh, you know, Python looks like they're
00:23:51.440
making progress getting rid of their gill, and and I hope at some point, uh, Ruby will will do the same.
00:24:00.159
Yeah. Uh, for me, 10 years, it feels like really long time because the
00:24:05.280
because Solid Q has been like one the 1.0 zero version uh doesn't even have a
00:24:11.919
a year. It's not even a year old. So 10 10 years salontentine. So I don't know
00:24:17.200
how much is going to change. It's still quite immature in some parts. So I I
00:24:23.760
will hope that it will mature and that some of the parts that I'm not super happy with especially around concurrency
00:24:32.159
controls that part I I'm not happy with the performance there. Um, I hope we
00:24:38.960
figure out how to improve that in future versions. And yeah, I just I suppose I I
00:24:45.120
will hope that it continues being praise default hopefully and and that we can
00:24:52.480
get it to a more like mature state. I I I guess and improving the
00:24:58.400
parts that aren't that great. it it's would it wouldn't ever be as performance
00:25:04.480
as performant as Sidekick or or Karafka or anything like not not at all. But at
00:25:10.799
least the parts that I know can can could use an improvement.
00:25:18.799
Yeah, I agree with Mike on on Gil on Raptors. I've been complaining a lot on
00:25:23.919
them for years. Opening issues uh being ghosted a bit as well. Uh yeah, we're
00:25:30.559
not there yet with them and uh I hope that well I really hope that in 10 years we will. Uh I would love for them to uh
00:25:39.760
start working as expected faster cuz I I don't think users code needs to be
00:25:44.799
bracking before and after the job execution uh
00:25:52.480
that could benefit greatly out of it. Uh I really hope that in in the context of
00:26:00.480
rails that we're going to get a abstraction on top of active job like active event something that is more
00:26:05.919
generalized to bring event- driven architecture to the rails community which has been in in my opinion mostly
00:26:11.600
entitydriven you know you have like a user.save user.update update. You don't have
00:26:16.799
batches. You don't have streams. Uh even with sidekick batches, people are just not used to elevating them in my
00:26:22.640
opinion. It's like, yeah, I just I'm going to spawn a job for every single user or whatever, right? Uh and in in
00:26:30.720
context of my things, I I really hope that in 10 years I'm going to I'm going to finish the show UI. Uh which probably
00:26:37.440
is going to look like Sidekicks. Uh which is how the first UI of Karafka
00:26:42.880
looked, right? I emailed you, can I steal the Sidekick's design? And I ask you, is is Sidekick's UI under LGPL or
00:26:50.960
or some sort of a copyright? And you just said, yeah, whatever. Yeah. All the code is LGPL in Sidekick
00:26:56.559
itself. Yeah. Yeah. So that's why it looked like that. Feel free to steal it. No, I did.
00:27:02.960
I did change it though. Uh after a guy that probably is here complained that it looks 2010. Uh
00:27:10.960
it served its purpose though. It's 2012. Thank you very much.
00:27:16.480
Yeah. Uh and in terms of uh Karafka, I I'm like 50% done with it. There's like
00:27:22.559
a lot of capabilities I didn't embrace yet cuz I haven't had time or uh they're
00:27:27.679
just less important than other things I've been doing. But uh yeah, I hope to bring many new capabilities. Kafka is
00:27:34.960
also evolving itself. uh the Apache team is working on cues on normal cues uh not
00:27:41.679
constrained by uh Kafka's technical like design limitations. So I hope to bring
00:27:47.279
that as well to the Ruby community. Yeah,
00:27:52.640
I think I'm really hopeful that active job uh gets developed. Um, when I
00:28:00.559
started Good job in 2020, um, like active job, like the active job
00:28:05.679
interface, the API, I think hadn't really been updated since it like was first released in whatever Rails 5
00:28:12.720
something. Um, so and like as I was sort of, like I said, like there's sort of I
00:28:18.960
imagined it being very simple and it has gotten a lot of features. Um, but these
00:28:24.880
are all things in good job and like I sort of from talking with Rails core people and I mean obviously we could
00:28:31.039
fill a stage with people with competing systems that active job itself is very
00:28:36.640
is sort of like the lowest common denominator of these backend job systems. And so I'm sort of like hopeful
00:28:44.559
now uh that especially with solid Q that we have like a firstparty
00:28:50.799
uh backend that active job will get more attention um and get a more interesting
00:28:58.240
API or potentially have like other features like concurrency actually built in. Um and I guess like I'll say I think
00:29:05.039
I'm starting to see that. Um, like I think there's some new discard hooks
00:29:10.640
that uh I think a Shopify engineer contributed. I saw continuations
00:29:15.840
uh that there's like some good features that are going to be I think in Rails 8.1. Um, and so like I'm starting to see
00:29:23.440
some of that ice break on active job. Um, and so like mostly I'm just excited
00:29:30.159
to see yeah more of these things where it's just like great for good job. I can just hook into make sure that I support
00:29:36.000
that rather than have to build it uh entirely inside of of my own back end.
00:29:43.360
All right, I have one more question before uh we're going to go to audience questions. Um, so I mentioned that like
00:29:50.480
we're able to fill a stage here with with four people that are all working on things that sort of are in the same
00:29:55.679
space. Um, where do you think there are opportunities for people who want to develop libraries with the same level of
00:30:01.760
importance as background job systems? Uh, so not within background jobs, but
00:30:06.960
is there anything that you wish somebody were to take uh and develop into a into
00:30:12.640
a library? No,
00:30:18.480
I don't know. I mean, if I if I had that idea, I'd probably build it myself. I mean, we're all engineers. Um, I I just
00:30:24.000
encourage people to scratch their own itch, right? That's that's why I created Sidekick because I I was literally the
00:30:29.440
first customer for myself. I wanted this system. Um, and so if you're if you're
00:30:34.960
using something and you see it as nonoptimal or you know a better way to build that mousetrap, um, do it. uh you
00:30:43.760
know in my case uh I had been doing open source for quite a while and so how to
00:30:50.320
how to build for the long term was very much on the front of my mind so uh
00:30:55.520
that's why I went the commercial direction and so you know you can obviously if you want to build something for fun in your spare time great but you
00:31:02.720
can also think about if you're building infrastructure that other people are going to rely on how are they going to
00:31:07.919
rely on you for the next decade uh what's going to keep you going Um, so,
00:31:13.919
uh, we each have our own motivators and so that that's where you really need to think about yourself and what motivates
00:31:19.200
you and what what would motivate you for the next decade to work on this project.
00:31:27.039
Yeah, it's a it's a tough question. I I think one thing that I not sure rails still
00:31:35.679
still has have has a good answer for is anything around monitoring and alerting
00:31:43.519
and that kind of thing like everyone does have to do their own and currently
00:31:50.240
like at 37 signals we we do a mix of Prometheus we use Graphana um we
00:31:58.640
have like different um ways to alert, but it's nothing that race doesn't help
00:32:06.399
in any way. I I wish there was something that you could just could just use. I
00:32:11.760
don't know if it's the same as maybe not the same as as a job back, but that
00:32:18.720
would be really really cool to have it provide by the framework.
00:32:23.919
Yeah, I agree with you like er error tracking generalized one that you could have in a database, right? Uh I think
00:32:30.960
there's a space for uh complex workflow tooling for Ruby and Rails, you know,
00:32:37.279
there there's like temporal there's a sidekick batches uh but they're not generalized. There's like pretty limited
00:32:44.960
I don't like the APIs of neither yours or acidic job. uh sorry but but I I
00:32:52.240
think that's uh you know we have our own expectations and I I think there's a big space there with being able to
00:32:58.640
understand how the workflows are executed having a really good de developer experience and I strongly
00:33:03.919
believe that if you if you build something that has a really good DX people will be first of all using it and
00:33:09.120
then at some point hopefully paying for it uh obviously there's the AI space you
00:33:14.880
know uh I kind of feel we're a bit behind it with with certain pieces of
00:33:20.559
tooling that we could nicely plug in and like utilize uh some of the quite useful
00:33:27.120
cap capabilities we could get out of that again with good developer experience and uh decent UI uh that's
00:33:36.080
and things related to async like I didn't mention it before but uh
00:33:42.480
stuff on top of what Samuel Williams is is doing for you know acing work that's like none of us embraced it yet. I I
00:33:49.120
feel uh and some of the things that LLMs require or how they work are actually
00:33:57.120
play really well with with the async reality. So I I think there's a huge space there as well. Can it be
00:34:03.120
commercialized? I have no idea. Uh because I strongly believe you have a you need to have a UI. Uh and only then
00:34:09.679
people like can justify the the expense. But yeah, but that that's my opinion. I
00:34:16.320
mean, I disagree with none of that because I think the the DX side um
00:34:22.879
having a nice UI like I think just like mentioning the the dashboard side of
00:34:28.240
like Sidekick as being really great that I think like you know I'm like what does good DX mean? um that I think a lot of
00:34:38.079
like good good libraries I really appreciate are ones like yeah that I can actually see it and use it and yeah
00:34:45.679
having a nice developer dashboard that's easy to set up like mount inside your Rails app. Um I'm going to keep pounding
00:34:52.800
on the I want to just run in a Heroku cheapo dynino. Um and yeah, especially
00:34:58.079
like uh this is me just asking for somebody to build this like the logging
00:35:03.760
um met logging metrics and exception monitoring just like that I could just
00:35:10.079
mount as an engine in my app put in my Postgress database um would be uh really
00:35:16.480
really nice um to have and I think those are the things too is just yeah the like
00:35:21.599
visibility into uh whatever the heck this library is doing uh is really
00:35:27.280
really valuable. All right, we are at perfect timing that
00:35:32.800
do we have any audience questions and do we have somebody that can run with a microphone to ask them
00:35:40.800
or even see because it's like Yeah. Do we have one more mic or should I
00:35:47.280
share mine now? Okay. No questions.
00:35:53.040
Yeah, there's one. Okay.
00:36:01.599
Um, since we're as a community kind of moving towards after commit as the way that we incue jobs, which makes sense,
00:36:08.240
especially for something like sidekick where you couldn't even do it transactionally if you wanted to. um what do you guys recommend for
00:36:16.000
uh kind of observability I guess you could say because obviously if you do something in an after commit you've kind of lost even any capability to do it as
00:36:23.440
part of that transaction or as as in the course of that work so like if it failed halfway through the work at least like
00:36:29.200
part of it has failed so if the after commit fails uh what are your recommendations for
00:36:35.440
like oh well that job didn't get incued maybe I got an error to century or whatever but if I don't catch that What
00:36:42.720
do I do? What do what do you what would you do in an application or what would you recommend?
00:36:52.560
I'm just going to agree with you. Okay. Um, yeah, that's a it's a good question.
00:36:58.560
I will say it depends on what the job is. Like I would say in many cases it
00:37:07.200
may not matter that the job didn't get in queued so you lock that. But it doesn't cause any data integrity
00:37:15.119
problems, right? Our problem was with jobs that you expected to you expect
00:37:20.800
them to run no matter what because otherwise your data will be strange like
00:37:26.240
it will be it will be not not right. So you will need to fix it afterwards. But
00:37:31.680
the many jobs are not like that. Many jobs maybe they they are like turbo streams jobs that if they they get
00:37:38.960
missed doesn't matter or other jobs that yeah maybe you may miss an email but
00:37:45.680
maybe it's not that important or maybe there is a way for you to detect that you didn't send it or yeah or maybe you
00:37:54.400
have some kind of for example for in in our case in in hey which is the app I'm
00:38:01.359
I've been like I'm most familiar with in our email service. There are certain jobs that will be
00:38:08.960
catastrophic if they don't if they don't run. So if they don't get incued or if they don't run for some like someone
00:38:16.000
pulled the plug on the server where the jobs is running and whatever and that is emails related to sending email and
00:38:23.040
receiving email. So we have alerts around that like we we track things in
00:38:29.119
the database that you can alert on like this mail has been delivered or this mail hasn't been you know pro processed
00:38:36.160
or things like that. So we have alerting there. So we know that if the job fail
00:38:41.200
to be incued, we'll we'll be alerted about that. But those are all very
00:38:47.760
specific like we know what jobs those are and we added that logic to like to
00:38:53.359
those alerts. And in other cases we even had custom roll back logic like we encue
00:39:00.480
the job but we when we incue the job we check if the job was incued and if it wasn't incued we roll back whatever we
00:39:07.520
did before. But this is we do only do this in one case and it's because it's very a very specific thing for our app.
00:39:15.200
So yeah, I would say it really depends. I will hope that in most cases losing a
00:39:24.000
job is not catastrophic like that that you can you have to go back and see what
00:39:30.160
you failed to in quue and fix it. And and for the rare cases that are this
00:39:37.359
critical then you will have to do something like alerting or manually
00:39:43.280
handling the situation in the where you the job.
00:39:49.680
I agree. Just just kidding. But for subcritical stuff like whatever uh because Kafka is
00:39:57.040
like absolutely distinct from the database, right? But for critical stuff there's the transactional outworks
00:40:03.440
pattern where you actually store the messages or or jobs or whatever in a database and you have a independent
00:40:09.440
process that is dispatching the data with in in like jobs or yeah whatever in
00:40:14.480
batches uh to make sure that something you mentioned does not happen. Uh it is
00:40:20.480
not brewed into Karafka yet the the transactional outbox pattern engine.
00:40:25.920
There are two available ones that you can plug in that are open source. Uh but
00:40:31.040
yes, I'm working on a third one. Uh exactly because I I I really think it is
00:40:36.480
something really important for people to be able to glue their data sources uh and tooling in a reliable way.
00:40:47.359
Yeah, there was something that I I wish a t job allowed which will be like maybe
00:40:55.040
during the within the transaction you could have like a sort of log somewhere
00:41:00.480
of the jobs you are going to incued do they will get incued after the commit but you will have something that will be
00:41:06.640
stored within the transaction um with some information like you could use maybe you could mark like certain
00:41:13.839
jobs as critical or whatever and it will be stored automatically in a in a sort
00:41:19.200
of log, rolling log, whatever that you could check afterwards and see if those
00:41:24.480
jobs actually run. Something like that. It would be nice. At least for us, it
00:41:29.599
would be nice. We could simplify some of our custom code to handle those situations.
00:41:39.200
Who has another question? I see a hand back here.
00:41:46.640
you. Um, can you talk about how you chose the name of your backend and
00:41:52.400
knowing what you know now, would you pick something else as the name? Uh, sure. Yeah. So, Sidekick,
00:42:00.880
uh, I named it because in any sort of Rails app, you always have the main app
00:42:07.839
server like Puma, but then you also have this thing also running on the side. executing your job. So, it's sort of the
00:42:14.319
sidekick to your main app server process. And that's just really what
00:42:20.079
what it came from.
00:42:25.680
Yeah, that's simple. I just put an R inside of Kafka because of Ruby language, right? And
00:42:31.359
it's like Karafka, which is Yeah. Karaf in in in in Polish. I was like, yeah,
00:42:36.880
that's a good name. Uh it also protects me against litigations like Apache owns the Kafka trademark. Um so any type of
00:42:44.720
commercialization would just go down and I would have to rename which is by the way a really important thing no one
00:42:49.920
thinks. Uh they they I don't want to say steal but they borrow the names based on
00:42:54.960
other languages technologies without being aware that people may have trademarks on them. Uh with Shyuken
00:43:01.359
obviously it was not me but sidekick is you know you kick someone shyuken you punch someone. So kind of kind of get
00:43:08.319
the inspiration. Yeah, it was also a I think it I I wouldn't name it that way because uh
00:43:15.920
Capcom tried to uh trade trademark it. Uh they did fail though. Uh the the
00:43:22.079
request was rejected. Uh but there's still like a bit of a gray area to have a name that is uh associated with
00:43:28.160
something absolutely else that you don't control. Right. So very quick little side note there. I I
00:43:34.800
eventually got the sidekick trademark, but I was rejected the first time because 20 years ago, T-Mobile sold a
00:43:42.960
little cell phone that they called the sidekick, and the trademark office said people
00:43:48.880
could confuse your enterprise software with a consumer cell phone.
00:43:54.400
W Don't Don't you specify the the like application space for trademarks like
00:43:59.920
software, games, other things? Yep. I specified software. They rejected it anyway. But like I said, the second
00:44:06.560
time around I got it. Yeah, mine was again super easy. I
00:44:13.599
didn't choose the name. No, in my case it was because we had solid cache uh
00:44:18.800
that had been built before um by one of my teammates and he was uh in in the
00:44:25.280
beginning that was called active record cache but we didn't like the name very much and I think it was DHH who came up
00:44:32.240
with the name for solid cache like because of the solid state disk and cash like you know and and it's supposed to
00:44:39.680
be solid and so when I was building solid uh solid Q it was
00:44:45.040
that I should be called solid Q because they go together. So super easy. Uh with good job it was very much like
00:44:53.200
uh defensively for my own psychological safety. Um of the entire point being
00:44:59.280
that like it's good, it's not great, it's not the best, it's not the fastest,
00:45:04.319
um it's not the most featureful. And that was largely from being on the internet uh and feeling and especially
00:45:11.440
with like background jobs. I feel like always the question is just like but is it the fastest like what is the
00:45:18.240
performance of it and you know like I say like my my goal was run on Heroku it
00:45:25.119
was like that's really what I wanted and so by naming it I was just like really trying to sidestep anybody trying to
00:45:32.960
like place it as like or make asking me
00:45:38.560
to place it at the pinnacle of being the best of everything else. And so that was
00:45:44.000
like always I was like ah it's it's just good. It's good. Um don't make it be more than that.
00:45:49.520
Well, just keep in mind you can sidestep but you cannot sidekick because he has a trademark on it.
00:45:55.040
Watch it.
00:46:11.839
Are there any uh patterns or practices from outside the Ruby community that you
00:46:17.200
either take inspiration from or you think our community could learn from?
00:46:24.960
Okay, I can start. Uh so my I've been accused of writing Ruby like Java. I
00:46:30.880
I've seen comments like this on Reddit. Yes, I've seen his code. It looks like Java. Uh I I see it as a compliment
00:46:37.200
actually. Uh I I think we're not but again me doing marketing partially so
00:46:43.359
I'm biased uh not embracing event-driven architecture enough uh entity based
00:46:49.520
telling other pieces of system what they what they're supposed to do instead of telling them what you did and making
00:46:55.440
them react independently. So that's something I I I've been missing in many of the systems I worked with. uh the
00:47:02.720
same with a complex workflow modeling uh which is done better in some other
00:47:08.640
heavyduty old-fashioned enterprise systems. Uh apotonic with trailblazer
00:47:14.640
tried I don't know where he is with it. Uh probably in a bar drinking beers. Uh
00:47:20.720
but other languages have tools for modeling including like visual modeling of workflows being able to debug things
00:47:28.400
and understand that and we don't have that. So from my perspective this is something that we could learn from and
00:47:34.480
obviously better concurrency right uh but that we all know that so
00:47:41.040
traditionally the background job has just sort of been this black box you know I'm executing your job I just call
00:47:47.839
perform and and then it it finishes um but one of the more recent changes has
00:47:54.000
been iteration support uh so now that perform blackbox has been broken into
00:48:00.640
a a number of smaller steps. Uh and that's I think been a benefit um to to
00:48:06.319
make more predictable uh jobs. Uh so that's one one pattern that I I've
00:48:12.560
been happy to see the community uh start to uh to embrace.
00:48:23.359
Yeah. I I'll just say I don't like I'm like outside the Rails or Ruby
00:48:29.040
community. I just really appreciate like this group here. Like I'll just say um like Mike has been really helpful. Like
00:48:36.640
Mike reached out to me two years ago um and was like wanted like you know it was
00:48:42.880
I think it was like co and like it was just like oh we should just like chat um about stuff and you were really helpful
00:48:49.200
with like talking about revenue models and you you know had an opinion that you wanted like to share and encourage me to
00:48:55.599
explore that and I did and I appreciated you taking the time with that and like um you know I think a lot of this where
00:49:03.200
I'm just like I guess my theme here is like less on the technical side, but maybe just being nice as a Ruby theme.
00:49:09.520
So, I'm not actually saying anything new, but like talking with Rosa and, you know, like uh having very nice
00:49:16.000
conversations with you about um Solid Q and you asking for like advice from me about good job and um I do feel like
00:49:23.520
maybe Mike your theme of this is a black box that you put your thing in. Um, I've really appreciated just like the number
00:49:30.720
of people that are willing to help um and like like want people to be
00:49:36.880
successful. Uh, and so like I do feel like that aspect of of um doing
00:49:42.480
libraries and things that uh there is a really good community. So yeah, this is not Ruby has a great community. So this
00:49:48.640
is nothing new. Yeah, this guy is great. I I launched my pro version and he just commented it's too cheap.
00:49:53.680
I am pretty awesome. It's true.
00:50:02.720
You're right. By the way, it was definitely too cheap at that time. It usually is the first price is cheap.
00:50:11.119
I don't see it. Uh so for the folks that are just like
00:50:19.359
active job based um although I guess psychic has an active job version and I think you said Karafka can work in
00:50:24.880
active job as well. Yeah. Um, do you wish Active Job was faster than it is?
00:50:30.480
Uh, I know it obviously works really well. It's fast enough for like pretty much every use case, but you can't help but see like the Sidekick benchmarks
00:50:37.119
that show that it's like multiple times slower than just raw Sidekick. Um, do you think there's anything to be done
00:50:42.720
there or is it just kind of ah, it's fast enough. Who cares? That's pure marketing probably on his side. So,
00:50:48.640
but he's awesome so I don't think he would do that. Lies, lies, and benchmarks.
00:50:54.480
Uh, yeah. I mean, very few things have taken more of my mental space than the
00:50:59.520
tradeoff between keyword argument support and pure JSON, right? That's the
00:51:04.800
thing everybody either loves or hates about Sidekick. Um, and I've always tried to square that circle and never
00:51:10.720
been able to do it so far. So, at least good enough that I would want to release it. Um, but that's hard. Um, I I want to
00:51:19.119
get to the point where it's it looks like a nice seamless method invocation and you can use keyword arguments and
00:51:25.359
anything else you want. Uh, I just haven't gotten there. Um, I've got a a
00:51:31.359
huge installed base and backwards compatibility is a massive concern and
00:51:37.520
so it's just hard to make any any large scale change to Sidekick and a similar
00:51:43.520
system uh after a decade of of installs. So, you know, it is what it is right
00:51:50.240
now, but that doesn't mean it can't get better a year or two from now. Keep hope alive.
00:51:57.200
I don't think it matters. um with the performance of it. I guess I feel like
00:52:02.640
always the micro like as soon as you put something in the job um it's gonna blow
00:52:08.079
out the performance um of it. So I guess like like that little bit of of invoking
00:52:15.119
the active job wrapper is going to end up being an infinite decimal amount of
00:52:21.599
time that's going to be happening. Um and I'll just say like I like keyword arguments. I like global ID. Um I like
00:52:30.559
uh I wish more people serialized more data into their active jobs. That's a
00:52:35.839
common question I always get is just like how do I put um the same data into
00:52:40.880
everything like current attributes or something. There's something that needs to get built into active job. Um is
00:52:47.119
taking the current attributes from the environment and serializing it into the job so that those current attributes are
00:52:53.200
current in the job. Um but like I think there's there's quite a lot of potential
00:53:01.599
um in active job that can slow it down. Also uh rails instrumentation is slow
00:53:09.040
and I think for sidekick there is less you get to side s side s side s side s side s side s side s side s side s side s sideep all of all of that like rails
00:53:16.640
rails does a lot for for good and for bad trying to be helpful and that that
00:53:22.079
takes time. Yeah. My primary use case is not running active job and convincing people not to
00:53:29.040
I guess. Um but as you said the the moment someone puts somewhere something
00:53:34.480
inside of this job it doesn't really matter that much. Um uh could it be faster? Most likely yes.
00:53:41.680
Does it matter for most of the people here? Probably not. Uh if it matters to some of the people only then I will
00:53:48.480
actually look into it more. Uh yeah, it's I think it's a it's a good job, you
00:53:54.480
know, it's like active job is a good job. So it's a solid job as well. So
00:54:00.960
it's it's not a side job for you, Mike. So sorry for that. Think we have time for one more question
00:54:08.800
if it exists.
00:54:16.480
I can't see anything. Yeah, it's really hard.
00:54:21.839
I think we'll probably all stay up here if anybody has questions and wants to just come up and chat. I'm sure I'll be
00:54:28.000
willing to stay here and I'm sure the rest probably would be too. Yeah. All right. Thanks everybody.
00:54:35.440
Thanks for coming. Thank you guys. Thank you.
00:54:40.720
Thank you.
