Search
RubyConf 2012
Service Oriented Architecture at Square
Edit
#service-oriented-architecture
#jruby
#microservices
#api-documentation
#deployment
#code-quality
#security
Summarized using AI

Service Oriented Architecture at Square

Chris Hunt • November 01, 2012 • Denver, Colorado • Talk

In his presentation at RubyConf 2012, Chris Hunt elaborated on Square's approach to Service Oriented Architecture (SOA) using Ruby and JRuby. He introduced the concept of SOA and discussed the challenges faced by Square's evolving architecture, particularly the transition from a monolithic Rails application to smaller, isolated services. Hunt noted that the initial monolithic structure had become unwieldy, with a significant number of controllers and models that slowed down deployments and made continuous integration cumbersome.

Key points discussed during the talk include:
- Definition of Services: Services are defined as isolated units of functionality that perform a single purpose well, allowing for better scalability and maintainability.
- Architecture Choices: Square employs multiple programming languages, including Ruby, Java, and Closure, all running on the Java Virtual Machine (JVM), which simplifies deployment across different languages.
- Creation of New Services: Hunt described the process of starting a new service—a payments map service—as being initiated with a boilerplate setup that includes code quality metrics, API documentation, and deployment configurations. Automation tools like FOC (for documentation) and Cane (for code quality checks) help streamline this process.
- Testing and Isolation: The presentation emphasized the importance of testing in isolation. Hunt introduced the use of a tool called Foreman to run fake versions of services, allowing developers to create local environments that simulate interactions without needing live services.
- Security Measures: Given the financial nature of Square's applications, robust security practices are implemented, including SSL certificates with mutual authentication to ensure secure communication between services.
- Monitoring and Logging: Post-deployment, Hunt highlighted the importance of monitoring applications and tracking errors using tools like Airbrake and Splunk, which facilitate error reporting and log analysis for improved operational oversight.

In conclusion, Chris Hunt's talk showcases Square's commitment to evolving its infrastructure to better handle the complexities of SOA through disciplined testing, deployment practices, and security measures. The main takeaways for attendees were the value of adopting a service-oriented mindset, the importance of robust documentation and testing, and the necessity of monitoring for maintaining application health.

Service Oriented Architecture at Square
Chris Hunt • Denver, Colorado • Talk

Date: November 01, 2012
Published: March 19, 2013
Announced: unknown

SOA is hard. Learn how Square is approaching this problem today with JRuby and where we hope to be in the future. We'll go from git init to cap deploy, covering Square's approach to testing and service isolation, dependency management, API documentation, code quality metrics, data seeding, schema versioning, logging, exception handling, security and password management, deployment and more.

RubyConf 2012

00:00:15.200 my name is Chris and our work at square and today I'm going to talk about how we make a new service today at Square
00:00:21.560 specifically a ruby service because we also have lots and lots of java services and we do those very differently so uh
00:00:28.039 Ruby um let me start first first by talking about Square so I'm on our uh
00:00:33.760 risk engineering team so we have a bunch of really small teams one of those really small teams with with uh two
00:00:39.079 Engineers that I work with now is risk engineering so we have because we deal a lot with money um there's a lot of risk
00:00:44.680 with that some people do really really large transactions and we want to make sure that we kind of vet their business a little bit before we just uh give them
00:00:50.559 all that money because it could potentially um be risky for us to allow that to go through automatically so the
00:00:55.960 team I work on we write tools to make that really fast and then we also sit with our analysts and we try to automate
00:01:01.519 um a lot of the work that they do so that they don't have to do as much work also so we don't have to hire 5,000 people to actually look at all the
00:01:07.560 transactions that come through Square so the square story kind of started um we
00:01:12.920 had uh well I didn't I wasn't at Square when Square started uh I've been at Square for 11 months but the idea was we
00:01:19.000 want to allow anybody to be able to process payments right so uh that's a
00:01:24.320 pretty lofty goal but it was really defined so we said hey cool let's make an app for iOS devices let's make an app
00:01:30.400 for Android devices and let's make uh an API for those to talk to so of course we
00:01:35.640 use rails for that uh and people started using it right so we have this one API that we're using everybody's taking
00:01:40.680 payments it's working out really well and then we're like well maybe we want to send people receipts now right so
00:01:46.119 then we had five more controllers for receipts couple more views appreciate you guys see where this is going because I'm sure Many Of You' had this problem
00:01:52.280 right so the next thing we want to do is make it so that users can log in and see their history of payments and sort them
00:01:58.159 and search them and things like that so we had 10 more controllers a bunch more apis in the same rails app uh and it
00:02:04.240 starts getting really really big so this is what it looks like today this is our like monolithic rails app we have 183
00:02:11.200 controllers we have 197 models and 657 line uh, lines of code so this takes
00:02:18.440 like an hour for our CI to run and every time you want to add a feature it's like at least two days before it's going to
00:02:23.720 end up in production because we got to run the CI once it fails you have to fix it then you run it again and then you
00:02:29.160 have to deploy and you can't deploy anytime you want you have to deploy in the middle of the night because people are trying to like run money through
00:02:35.080 your system right so this is kind of the reason why we try doing small Services why we're trying to do that today rather
00:02:41.080 than adding features into this existing app that we still use uh but we're trying to pull stuff out of here cool so
00:02:48.640 I I I'm using impress.js for this so I'm going to do a lot of like animations because they look really cool so
00:02:55.480 this I'm going to start just by Loosely defining what service oriented architecture is I'm not going to try to
00:03:00.720 talk too much about it because this is not one of those conferences but this is a service so it's it's it this is kind
00:03:07.599 of what a service should be it's just a little sphere that you could move all over the place it has really defined functionality it's exactly like you
00:03:13.560 might do objectoriented programming you want to define a service so it does one thing it does one thing really really well whenever you have another feature
00:03:20.040 you want to work on if you're not going to if if you're trying to decide should it go on this service should it go on this service should it go on this
00:03:26.000 service generally you just make a new service unless you really really really think it relates to uh what's already
00:03:32.439 there um another nice thing about Services is um because they're individual pieces right you can put one
00:03:38.319 in one data center over here you can put one in one data center over here or you can move them around you can put them on
00:03:44.239 the same H different Hol like you put them anywhere you want because they're isolated so in this case I have uh my
00:03:49.920 first service and then I have a second service on a different slide and we see the transition there as well so the
00:03:55.439 little dots the little dots uh are kind of like the that interface that we would
00:04:00.680 use um to talk between services at Square we mostly use Json um that's
00:04:06.360 worked real real well for us uh other people might use things like a a messaging system where you have one
00:04:12.480 giant queue and you just kind of throw all your messages on there and everybody can look at them and that's that's really good too but uh we are not doing
00:04:19.799 that um maybe we will someday cool um so now we kind of know what services are um
00:04:26.199 the way that we deploy our services is uh we use Java um there we go so we use
00:04:31.280 Java so the reason we've decided to use um Java is kind of like our infrastructure is because we don't just
00:04:36.680 do Ruby we also do Java um we also have some closure stuff and we also have some python stuff and all that stuff runs on
00:04:43.360 the jvm so we use J Ruby when we use Ruby um we use Java when we use Java um
00:04:48.880 if we were to use Python we can use something like jython I don't know I don't think we actually do that but I mean we could um so the nice thing about
00:04:56.520 having uh one infrastructure for all your apps is you just start up a new app container and you can put any of your
00:05:02.199 services on it it doesn't matter what language it is because they all run on Java so the only requirement for a new
00:05:07.240 host if we're provisioning a new host is that it has the jvm and then we can stick any one of our services on there
00:05:12.600 because they all run on just the jvm and the way that we deploy them is that's the only thing they require is the jvm
00:05:19.000 cool so now that we kind of know what services are and why you might want to use them we're going to pretend like
00:05:25.319 today is your first day at Square congratulations guys and you're going to start working working on a new service
00:05:31.720 and this is going to be a uh payments map service
00:05:37.800 go oh there we go okay so what we want to do is we want to make a service that
00:05:43.639 shows a map of all of the user payments so a good use case for this might be
00:05:49.120 some kind of receipt maybe when you log in you want to see all your payments we don't actually have this service so I just kind of made it up so I don't
00:05:55.400 really have a good use case for this but the idea is we want to have a user and we want to be able to show their payments so we're going to go through um
00:06:02.120 the steps that we might take to build this if we were at Square today uh using the tools that we use so Step One is
00:06:09.479 rails new right okay so not exactly but very similar to that a couple a couple
00:06:14.840 months ago um some very smart Engineers smarter than me went through and they typed rails new and they set up all this
00:06:22.000 stuff in that app that does nothing um they documented it they added code quality metrics they tested it set up
00:06:28.759 the testing frame this is all boiler plate stuff that you usually do for a new service right um they set up the
00:06:33.840 deployment infrastructure for it so got it all set up so you can do like a cap deploy for example they set up
00:06:38.880 monitoring and then security um for that service and all the other services is going to talk to so this app actually
00:06:44.840 does nothing but it serves as a template or a starting point for our new service
00:06:49.919 so if we actually want to start our service now we're going to use J Ruby and for the reasons I talked uh before
00:06:55.560 we're going to be deploying on Java that's why we're using J Ruby um so the command we would run is make me a rails
00:07:02.039 payment map so make me a rails is that magic uh template app I talked about this is just a script and I'll show you
00:07:07.639 in a second what this script does it's it's actually really simple but then we just go in there and we get a nit and
00:07:12.680 this is a empty rails app ready to be deployed we can deploy it right away so what does the script do search and
00:07:19.639 replace all we do is build a template app we search for template app constant and we replace it with our new app
00:07:25.319 constant we do it twice that's it so you can build this to just make a new rails
00:07:30.599 app get it deployed and then save that off as a good starting point um for all your
00:07:38.800 services all right so have you guys heard of Professor Farnsworth at all he's from future Rama he's like a very
00:07:45.479 like Ecentric crazy person um we're going so the tool I'm going to talk
00:07:50.680 about here is called FOC and it's named after Professor farmsworth uh I didn't name it but I know who Professor
00:07:56.240 farmsworth is so um this is something we made it Square to to make documenting really easy so generally if we're going
00:08:02.199 to make our new payments map API we want to start with uh documentation first and
00:08:07.280 then we'll write our tests and then we'll write our code so I don't know if documentation driven development is a thing um but that's kind of how we try
00:08:13.360 to do it and the nice thing about FOC is if you do not document your API in the way that you're using it in your tests
00:08:19.360 your build is going to fail so it kind of just reinforces itself if you don't have docs you're not going to get your
00:08:24.440 stuff deployed because the build isn't green um Okay cool so let's look at that pays API was talking about this is how
00:08:31.560 we're actually going to get those payments so this is something that FOC would um generate for us um based on
00:08:38.519 what we did so looking at this it's pretty self-explanatory we have at the top the URL we need to hit to get our
00:08:44.120 payment data we have an example request which just has a user ID and a limit and
00:08:49.240 then below those parameters are described and we can see the type an example and whether it's required or not
00:08:55.320 and then FOC also shows us an example response so we can see we're getting get a success Boolean which is probably
00:09:00.760 going to be true hopefully in most cases and then we get a array of all the payments so now we know uh how this API
00:09:08.360 works we know we just need to give it a user ID and we're going to get back some payments so now we can start writing our own documentation for our new API for
00:09:15.800 our payments map so now we're going to go through how to do that again like I said we're using FOC so here's the uh a
00:09:23.600 pretty vanilla payments map controller spec and all we need to do is that magic sauce line right there where we include
00:09:29.600 FOC spec Watcher and that's going to look at all the requests you make in your controller spec if it's not
00:09:35.760 documented then it's going to break the build um something you can do with FOC
00:09:41.760 is you can run your specs with the scaffold environment variable and that's going to generate documentation for you
00:09:47.320 um but it's very it's very empty documentation it just has a whole bunch of question marks so you need to fill these things in for example describe the
00:09:54.440 uh payment map uh Service uh that's that top describe block there and then we have the param all the parameters that
00:10:00.600 have been used in our specs and so that we need to describe those as well in this one spec we only passed in user ID
00:10:06.959 so it saved that off as the example there so we just need to describe it and say whether it's required or not and
00:10:12.440 then it also recorded our response so we can describe what the response looks like we have the example URL so in this
00:10:18.360 case it looks like we're just going to be returning uh a path to the image that's fine um but we do need to describe and
00:10:26.120 say whether that's a required uh response or makees sense um from this um uh at the
00:10:33.839 top you see the file name the file name describes the request so it's going to be it's going to match the path so this
00:10:39.560 is going to be available at payment slow with the method get and this will be compiled into that HTML that we just
00:10:46.160 looked at um fdoc will do that as well do the the compilation so this is available uh
00:10:53.079 online as well uh GitHub so I'm going to actually get a drink of water so memorize the URL
00:11:12.240 cool so now we've got our API documented and like I said we're going to do uh documentation driven development right
00:11:18.160 so now it's not wrer tests uh and our test is uh very very motivated by this
00:11:25.360 book I'm sure many of you have read this book but if you haven't you should check it out um something it really talks about is unit testing and and testing
00:11:31.880 your things in isolation so with Services that's especially important because on my laptop for example I can't
00:11:38.160 even run some of the services that we have so it would be impossible for me to test using those actual services so a
00:11:45.720 really good way um to do like your acceptance test and not actually have to talk to all of your service other
00:11:51.839 services is to make fake versions of them so this is an example of a fake
00:11:56.920 version of that payment service we looked at it's super simple uh we see we have a payment service module and a fake
00:12:04.160 server that's Sinatra and then we just describe that endpoint and we can even stick in the example response that we
00:12:10.480 see in the documentation but this will allow us to use a cool tool called foran
00:12:16.440 to boot up all of our fake services on our development machine so that we can write our tests and write our acceptance
00:12:21.880 uh and and do actual coding and um you know try it out so uh let me show you how to set this up with foran
00:12:29.399 uh here's the rack up file which this is going to be the same for every single service you just require your fake
00:12:34.480 server and mount it to the rout um and then you can take this rack up file and
00:12:39.519 include it in your proc file so lots of files here proc file is what Foreman looks at to start your services so Forin
00:12:46.560 is going to start two services for our new uh map application it's going to start rail server which is obvious we
00:12:53.120 want our map server to be running but it's also going to start the fake version of the payment server and it's
00:12:58.440 going to start that on Port 3001 you also notice uh on that Top Line there with the server we're starting it with
00:13:04.399 this weird isolate environment variable so the reason that's there is so that we can allow ourselves to actually use real
00:13:10.480 Services if we want to so inside of our rails app we'll have probably a yaml and
00:13:16.560 we do use a yaml of all of our services that we depend on and their URLs so the
00:13:21.639 payment service in production will be at this URL the payment service in staging will be at this different URL and the
00:13:27.440 payment service in development will be at this URL but if it's set to isolate then we actually want to look at Local
00:13:32.760 Host so every time we start forming we'll use the fake server um if we don't use foran uh then we'll be talking to
00:13:39.440 the real server which will live what wherever we Define on this um yamama which will probably be on our
00:13:48.120 laptop and that's how you start forming this will actually this is actually a lot like when you start rail
00:13:54.199 server and it starts showing the log files this will show all of the logs for all of the services you've you've
00:13:59.680 defined in your proc file and they'll all be prefixed with the name of the service so if you have like five different Services you'll see five
00:14:05.480 different colors of it's actually really you should try it out it's really cool um so this is where you can get foran
00:14:10.800 this is not something we made at square but we use it a
00:14:19.000 lot all right so next up uh we've gone to testing so now we're going to
00:14:24.320 actually start writing our code and something that's really important that we found when making all different Services is keeping some kind of
00:14:30.800 consistent code quality among them because you're probably going to have like two people over here working on this service two people over here
00:14:36.440 working on this service and they're going to have vast different opinions on you know little things like line length
00:14:41.800 and empty white space and all that kind of stuff right so a a a good way to kind of just take that out of the equation is
00:14:47.759 to Define that in advance and use a tool like cane this is Kane um this will
00:14:52.800 break your build if you violate Style Guidelines that you've set so I'm actually really really into the less
00:14:58.399 than 80 character thing so this is like a godsend because if somebody does a poll request and it's like 90 characters
00:15:04.480 long I don't have to passive aggressively say well maybe we should shorten the line all I got to say is the the build broke right so it's way easier
00:15:10.240 for me like little things like that are huge another good example is if you're
00:15:16.279 working on um a ruby 1.9 project maybe you don't want to use hashrocket syntax right you want to use the new fancy uh
00:15:22.639 Ruby 1 n syntax there's not really a good way to enforce that but it you can um build a rule in cane to enforce that
00:15:28.399 for you so if somebody does a PO request with um using the hash Rockets then the build's going to break and they'll have
00:15:33.519 to go back and change that uh cane is also available and
00:15:39.800 that's where you can get it I'm also going to try to make these slides available later if you don't want to write these down but they're pretty
00:15:47.079 self-explanatory okay so now we've got uh I didn't really put the code for a payments map in here because that's not
00:15:53.079 really interesting so we kind of just coded and now we're going to deploy our new service
00:15:59.519 um there's a bunch of different ways you can do this and this probably isn't the best way but this works really well so we have a uh Java that we're deploying
00:16:06.560 to so we need to use a Java web server right so the one we've decided to use is Jetty because it's really fast and it's
00:16:12.720 really small and you can actually kind of embed it into your rails app you don't need to assume that it's already on the host you're deploying to so again
00:16:19.240 it kind of serves that purpose of of being able to provision a new host right away and then put your service on it and not really worry about what dependencies
00:16:25.519 are there you can kind of just stick Jetty inside of your rails app um a really uh good tool for this and the one
00:16:31.120 we use to use Jetty is called jetpack so jetpack actually does all this for you you just need to create a config um Jet
00:16:38.399 pack. yaml and in here you can Define all the things for your app so here are some examples where we have um this is
00:16:44.800 the user on the machine you're deploying to the user that's going to need permissions to start this Jetty web server um this is where it's going to
00:16:51.399 live right so we want it to live in user local apps payments map wherever we deploy this thing to and then we can also specify things like the ports HTTP
00:16:59.160 Port https port and then our Ruby version so this is obviously an old gaml we're using Ruby version uh 1.8 but you
00:17:06.439 can use 1.9 and I think with uh J Ruby 17 that just came out 1 n is like the
00:17:12.360 default so that's pretty cool um so with uh again so this is Jetpack how how this
00:17:17.959 works edit the config file and then that's pretty cool there we go run
00:17:23.799 jetpack in the current directory it'll package it all up um and what so what this jetpack does is it downloads the J
00:17:30.799 Ruby runtimes from the J Ruby site or wherever you tell it to by default it'll be the J Ruby site um it'll download the
00:17:37.799 uh Jetty run times it'll uh bundle your gems in J Ruby and then vendor them uh
00:17:43.360 and it puts all that in the project oh Jesus it puts all that in the uh project directory and then when you do um your
00:17:49.640 cap deploy all that needs to do is bundle up your or tar up your directory and SSH it up um because everything else
00:17:56.280 is already set up Jetty already knows how to start your server your ports are already ready to go all your gems um J
00:18:02.000 Ruby itself that's all in the package so you don't need to even assume J Ruby is on your host all you need is Java you
00:18:08.240 run jetpack in the current directory and you're good to go um jetpack is also available uh at
00:18:15.200 Square jetpack and there's a a pretty good read me um to get you started because I probably totally did not cover
00:18:20.919 something on here nice so we're at the point now
00:18:26.559 where we're um deployed and there was a lot of voodoo in that deployment so feel free to ask me questions later um but
00:18:33.159 it's up there so now we need to monitor it right so I'm sure a lot of how many people are using this is the like the
00:18:39.400 airbrake logo how many people use air bra now like a lot how many people use hoptoad are they're the same thing so
00:18:46.440 okay so we actually use hoptoad but I'm um saying we use air bra just because it's the new hotness but it's actually
00:18:51.559 the same thing so what airbr does uh you configure your rails application with uh you just stick an initializer in there
00:18:57.760 with your API I key uh and maybe some other info but it's really really that's really all it is so I don't really need a code sample on here and then all your
00:19:04.440 exceptions are sent to air bre uh and this will email you if there's an exception it'll kind of track exceptions
00:19:10.520 over time so you could find things that are kind of becoming more problematic than they were uh it'll also let you comment on exceptions so it might look
00:19:17.080 you know typically something like here's a timeout error these happen all the time um there you are uh so this is
00:19:22.799 really useful for us uh for like interrupt driven triaging if you're it'll tell you if something breaking
00:19:28.840 it's not really good for a monitoring tool I don't think but it's good for interrupt driven stuff um to warn you if
00:19:34.880 something's about to happen another really awesome tool this has um nothing to do this this works
00:19:42.640 completely different than air bra this is more of a monitoring tool um cubism DJs you can make really really cool
00:19:48.559 dashboards with this how this would work is you take all your critical metric like Fu bar and Fu plus bar and Fu minus
00:19:55.760 bar but probably more stuff like in squares case we like to monitor um our
00:20:00.880 our payment apis and make sure that payments are actually going through uh if this graph starts looking really weird like we're getting a bunch of 500s
00:20:07.200 on our API host or we're running out of memory or our CPUs bottle rocketing uh uh that was really cool um
00:20:14.520 or our CPU is like going through the roof this is a really good tool for figuring out um that we need to start
00:20:20.400 looking at something so this is you'll probably start seeing uh activity on on here before you get exceptions we've
00:20:26.400 noticed um cubism s can take information from graphite it can take information from um Cube which is similar to
00:20:34.080 graphite but not but you know what I haven't actually used Cube um but we use it a little bit at Square I haven't used
00:20:40.120 it U very useful tool this is available on GitHub as
00:20:47.720 well okay so now we are at the point where we are deployed and we're monitoring our app Splunk is the tool
00:20:55.400 that we're going to use to figure out what's breaking right so one of the huge problems with having Services all over
00:21:01.799 the place is you never know where to look for your log files so if you're getting an exception uh even if you
00:21:07.280 think that so say for example we have five five API hosts that are taking
00:21:12.640 requests from uh handheld devices right and we know there's an exception but we
00:21:18.159 have no idea which host it's actually on um Splunk is a really good tool for that
00:21:23.799 because you can search through all your log files in one place so all your SE Services send their logs to Splunk and
00:21:30.679 then you can search for say a payment ID and you'll be able to see that payment go through all of your services and you
00:21:36.320 won't need to SSH into each box individually and look at the log files you can also generate really cool graphs
00:21:41.679 like this um that kind of break down uh whatever you want so this would be different different types of queries
00:21:47.360 stacked on top of each other across time so like successful versus failed payments that kind of thing um so here's
00:21:53.360 an example of a query that I use uh when I'm on call
00:21:59.000 um because what this will show uh for all of our API hosts any process that
00:22:05.200 involves payments tell me which ones are 500 in so uh and then at the very end uh
00:22:10.480 we got um count by action so Splunk is able to parse your logs and detect the
00:22:15.760 controller and the action any other thing you can kind of teach it what you want it to know it knows rails by
00:22:21.039 default but if you have your own custom log file you're going to need to probably teach it um but you can do
00:22:27.360 intelligent is like this so excuse me so this would show you like a graph of all the actions that are failing and so you
00:22:34.279 can kind of gear your attention towards the those specific actions in your controllers that are causing the most
00:22:41.480 trouble okay um uh cool so this is the
00:22:47.159 last thing I'm going to talk about and this is security between services and I probably have more details about this
00:22:52.799 than anything else um because it's it's more interesting in my opinion as far as looking at code so a lots of people with
00:23:00.159 internal Services don't really care about security right because it's behind your firewall who cares nobody's going to be able to get to your services
00:23:06.520 anyway we trust our developers we don't we don't think that they're going to write malicious Services uh but when
00:23:12.080 you're working with money you have like certain requirements um that you just you might as well just secure everything just to
00:23:18.720 be safe uh and it's actually really easy to do so why not do it so for security
00:23:23.799 we use um uh SSL certificates uh and we
00:23:28.840 just use we do what's called Mutual authentication so I'm sure everybody in this room well most of you in the room
00:23:35.080 probably are familiar with um setting up like an SSL connection uh for like if you're making a secured website like
00:23:41.880 Bank login or whatever you want to make sure that that session secure so you can't see the traffic but you're usually
00:23:47.520 that's that's a one-way thing you're not actually verifying like the user's browser for example making sure that they say who they say they are you are
00:23:53.400 with a password but you're not doing it with SSL uh with Services we can do that because we have complete control over
00:23:59.120 the services so every time we generate a new service we also generate an SSL certificate for that service and then on
00:24:05.919 the SSL certificate there's a little key called organizational unit which you can put whatever you want in there so we put
00:24:12.360 the name of the service in there so basically we have uh for our payments map service to go back to that example
00:24:18.240 we would create an SSL certificate for that we'd sign it with our Square private key um just to just so that we
00:24:24.400 can verify that that is a service we made and then we would set the organizational unit of that certificate
00:24:29.960 to payments map so now on all of our services when a connection comes in we can look at that organizational unit say
00:24:36.919 okay this is the payments map is this payments map allowed to access this API if it isn't then we'll return like a 403
00:24:43.640 uh forbidden uh if it is allowed then we'll just let the request go through so we can we can do like a per Service uh
00:24:49.880 kind of uh uh security there so I'm just going to go through like in a rails app
00:24:55.440 how you would do that like how you would get the organizational unit out of that certificate and then uh how you can can
00:25:01.159 check on that pivot on that and I'm going to take a
00:25:08.919 drink so where you would start is making some kind of Access Control list so we
00:25:14.039 just use a simple Json file for that so in our payments app um this is what the ACL might look like uh we would have uh
00:25:21.960 our resource which is SL payments we'd have the method get and then we'd have a list of all the clients that are allowed
00:25:28.440 to talk to this resource so in this case just payments map so as long as we have't have this in here we know that
00:25:33.840 our payments map is going to be able to talk to our payment service so now we'll go through kind of how to actually use
00:25:39.600 this ACL in the rails app because right now we're not using it for anything so the first step is to have
00:25:45.520 some kind of we're just going to use a API controller for this example so all your API controllers can extend this API
00:25:53.399 controller I don't know if you're into inheritance but for this example we're going to use inheritance um you can go
00:25:58.559 and uh Define a before filter on here and all this before filter has to do is say if this request is authorized Let It
00:26:06.159 Go through otherwise return a 403 and an aess denied so just block the request so
00:26:11.880 now anything that extends your API controller is going to be protected by that ACL so there's a lot of magic happening in request that and less
00:26:18.960 request authorized so we'll go look at that and see what that does so in request authorized you see
00:26:27.000 we're going to have the if let's start by looking at that bottom organizational unit um I was trying to think of the best way to explain this it's kind of
00:26:32.799 hard to go on without seeing all the code and we don't have enough room on the screen so let's look at organizational unit this is how you
00:26:38.880 actually get the uh OU as it's called out of the SSL certificate when the re
00:26:43.960 request comes in you grab it out of the environment you can you can pass in the certificate as SSL client CT you then
00:26:50.919 create an open SSL certificate using that raw that you got out of the request and then that's going to let you easily
00:26:56.840 access those things so then you can search through your certificate for a name that matches OU which is going to
00:27:02.120 be the organizational unit and return that so this will just give us that payments map uh string that we've saved
00:27:08.039 in the certificate and then if we look at that top method up there we'll take that result in this case payments map
00:27:14.960 and we'll take the request method which in this case is going to be get and then we'll also take the path info which is
00:27:20.360 going to be SL payments so we take those three things right the organizational unit which is payments map uh SL
00:27:26.480 payments which is the path and the request method which is get those are exactly what we Define in our ACL right
00:27:32.480 so then we take those three things and we look at the ACL for something that matches that so we find an entry that
00:27:39.600 matches those two things and see if the current OU is in the allowed
00:27:44.960 list so while you look I'm going to drink more
00:27:50.440 water cool so like I said uh that would be the end so we now have um security we
00:27:57.840 talked about uh getting started setting up all your stuff uh including documentation testing how to get it
00:28:03.399 deployed uh and then taking that off and putting it into a template app and then we talked about how to monitor the app
00:28:08.679 once it's actually up there and then do security between the different services so I can probably take like I mean I
00:28:15.600 have lots of time but I can take maybe just a couple questions more because I'm scared
00:28:22.000 yeah so the question was what do your clients look like so like the IOS and Android clients
00:28:30.760 oh I'm sorry then can you repeat the question got it okay so the question uh
00:28:36.000 not not like the handheld clients the question is uh how are your consumers of these Services kind of getting getting
00:28:41.240 out the data are they using Json are you doing or is there some kind of defined client that you're using is that right
00:28:46.600 okay so generally we just treat any client service like another service so we create an SSL certificate for it and
00:28:53.279 we go through the same apis and most of our apis are Json apis
00:28:58.519 so like a client you might make we would just treat that as a new a new service and give it and give it access in the
00:29:04.519 ACL and talk to it using uh the same Json API that we would use for service
00:29:09.559 to service does that make sense I want to talk to you later because I'm sorry your questions over my head but I will
00:29:15.600 talk to you afterwards yes sure yeah so the question is have you had any luck uh
00:29:21.360 to paraphrase chopping up your giant rails app into into smaller services and so like I said like that the screenshot
00:29:27.399 I showed earlier of the lines of code and stuff like that is how it is today um but it did used to be bigger and as
00:29:32.440 we work on more things we are trying to chop things out so a really good example uh is our loog the logged in portion of
00:29:38.360 our website that used to be a part of that main rails app you know just another controller in there you log in
00:29:44.039 you go to this controller you see this view you can manage your stuff um but we decided we wanted to redesign that that
00:29:49.799 portion of the uh app so when we did that we just made a new service so usually what we're doing is anytime
00:29:55.399 we're going to redesign something or just do some kind of large refactoring or really work on any part of the app do
00:30:01.159 a lot of work then we'll probably just do that work in a new service and then um use our um I guess we don't need this
00:30:07.399 anymore uh use our um large app that still exists to redirect to that new service uh and then eventually just chop
00:30:14.880 it off so um does that answer your question usually only when we're working on it we haven't actually gone back to
00:30:21.720 to mess with um stuff that we're not currently trying to change yeah that's a good question
00:30:27.360 question so the question the question was uh do you have just really really small servfaces or do you have that do
00:30:33.000 really specific things or do you have like a bunch of larger Services is that is that what is that what you're like the dependency between the services and
00:30:39.799 oh I see what you're saying yeah like some deep call stack right that you like service a call service B call service C call yeah so uh no we don't have that
00:30:47.440 right now um because that's usually like that that would almost be like a smell right um because you have if you have
00:30:52.640 one service that needs to call it's kind of like if you have it's the same way like if you're writing doing object or programming if you have one object
00:30:58.799 that's calling into like the 18th attribute of this other object then you should probably just wrap that in
00:31:04.120 something else so that's kind of that's kind of what we're doing I mean that's what we try to do we the the deepest
00:31:10.399 like call stack I could think of is maybe two um yeah okay one more question yes uh on
00:31:19.159 the front left okay good okay so the question is uh I talked about doing um
00:31:26.720 service isolation using something like Foreman to create a fake version of the service how is that different from making like a a mock like an are you
00:31:33.200 talking about like an rspec or something oh okay sure yeah so our
00:31:39.200 approach is like it's very simple like you described we're always going to get the same response back in in some cases
00:31:44.440 we actually have some logic in the fake service so if it's supposed to do something like um approve a payment or
00:31:49.679 something then we'll actually take the the payment that comes in we might actually in our fake service we might actually be doing a lot of the logic
00:31:55.919 that's in the real service we'll read it up switch something and then return it back um we won't usually persist any of
00:32:01.519 that data but there might be some logic in there um generally when when you're writing a fake version of your service
00:32:07.760 you test it um with the real Service uh and we put it in a gem so with the payment service for example all of our
00:32:13.760 tests that test our payments API will also test our fake endpoints and then
00:32:18.919 that'll be in a gem so I wouldn't actually have to write that fake service I would just include the payments Gem
00:32:24.240 and then that fake version of that service is available in my tests that way I don't need to worry about you know if typically if I'm talking to like 10
00:32:30.320 apis I don't want to have to write the whole payment service over again every single time I make a new app um so you
00:32:36.360 would just kind of test it um the same way U but yeah that often the data is like static is we return the same data
00:32:42.080 all the time for the fake service and then for for full your other question was do you do acceptance testing uh for
00:32:48.840 all your services like all together uh no we don't so other than doing
00:32:54.480 acceptance testing against the fake versions of them we test them by the by themselves
00:32:59.880 completely and then against fake versions and that's that's pretty much it we just assume that everything works
00:33:06.480 and then the the conversation between doesn't actually get tested okay I think that's it please
Chris Hunt
@chrishunt
Explore all talks recorded at RubyConf 2012
+46