Search
Shota
Parallel Testing World
Edit
#parallel-testing
#ruby
#performance-optimization
#concurrency
#rubygems
#testing-strategies
Summarized using AI

Parallel Testing World

Shota and Fukumori • September 29, 2011 • New Orleans, Louisiana • Talk

The talk titled "Parallel Testing World", presented by Shota and Fukumori at RubyConf 2011, delves into the implementation and benefits of parallel testing in Ruby's unit-testing library, Test::Unit. The speaker begins by addressing the importance of faster test execution in the software development life cycle, emphasizing how parallel testing can significantly improve development speed through faster feedback loops.

Key Points Discussed:

  • Introduction to Parallel Testing: The concept of parallel testing is introduced, showcasing its advantages when compared to traditional sequential testing methods.
  • Benefits of Parallel Testing:
    • Faster Test Execution: Parallel testing enables multiple test cases to run simultaneously, drastically reducing the total time required for test execution.
    • Improved Developer Productivity: Developers can get quicker feedback on their code changes, allowing for more effective test-driven and behavior-driven development (TDD/BDD) workflows.
    • Case Studies: A real-world example is cited where a popular testing library in Japan experienced improvements in test times, highlighting the real impact of parallelization on performance.
  • The Parallel Test Patch: The speaker details the history and technical workings of the 'parallel_test' patch committed in February 2011:
    • Describes how the patch works, utilizing a master process that manages worker processes to execute tests concurrently.
    • Explains the communication between master and worker processes using standard input/output methods.
    • Outlines challenges faced during implementation, especially regarding compatibility with different operating systems (notably Windows).
  • Performance Improvements: Several graphs illustrate the performance enhancements achieved through parallel testing, where historically long test suites saw considerable reductions in completion time.
  • Conclusion and Call to Action: The talk concludes by reinforcing the importance of adopting parallel testing practices and encouraging ruby developers to leverage the new capabilities introduced by the ruby 1.9.3 release.

Parallel Testing World
Shota and Fukumori • New Orleans, Louisiana • Talk

Date: September 29, 2011
Published: December 12, 2011
Announced: unknown

Ruby (MRI) has an unit-testing library, 'test/unit'. This is used for ruby's `make test-all`. In February 2011, I committed a patch ("parallel_test") that allows us to run multiple Test::Unit::TestCase-s at the same time. Because of this patch, ruby's `make test-all` can run tests faster. This patch also affects to existing tests using `test/unit`. In this talk, I take you into the parallel testing world. We'll talk about general parallel testing, benefits of parallel testing, multi-threaded or multi-process, etc. Then talk about the patch, the story of making the patch and describe how the patch works.

RubyConf 2011

00:00:17.199 okay hi i'm javan
00:00:24.800 is it subtle gay yeah i'm a shirt of kumoi and i'm a livi
00:00:30.800 comida i'm also known as solar
00:00:38.480 this is a committed ranking of levy repository above today
00:00:46.879 i'm here i'm getting down and this is the party monster also known
00:00:53.760 as knob is here i i uh i think and i know anyone can uh
00:01:01.920 can't beat him and you know and where's max
00:01:09.040 okay there's nobody simply someone of committers wrote that
00:01:16.560 touched by yukihiro matsumoto into a changelog and committed
00:01:24.080 and this is my uh my twitter please follow me but please note that i'm tweeting
00:01:30.960 about uh 300 tweets in one day uh someone said
00:01:36.079 this help someone uh you are doing doing uh twitter so much uh
00:01:44.079 but my timeline is uh i love your tweet and she is salami uh she's first name is
00:01:51.280 similar to my nickname but she's not me you know
00:01:57.600 what there's available here okay uh today i'm gonna talk about first
00:02:03.840 thing but before starting the power talk i have to say this thing but
00:02:09.280 you know my english skill is unknown so please ignore uh however i said
00:02:15.360 song strange included english next if you want to listen to general
00:02:21.280 news around my ruby 1.9.3 and the future of lily i strongly
00:02:27.840 recommend you to move to room 2 right now
00:02:32.959 i want to listen to though that but i can't
00:02:38.160 because it's conflicting with my talk it's myself to solve
00:02:44.800 last uh all will be in this talk means uh max will be implementation
00:02:50.080 also known as the silvery okay i missed the scheduling of uh
00:02:57.519 talking at luvigagi i think it will not happen today but i missed it athletic player flipping
00:03:04.640 time scheduling to play for the stock i've loaded all at uh slides
00:03:10.800 of this uh at all at yesterday 10 pm and i added some slides today because
00:03:21.360 i think i left some time to
00:03:26.400 sometime or some left many times
00:03:33.120 and after i ended my slides all over okay today i'll talk about these things uh
00:03:40.400 wi-fi testing and multi-slide mouse process and testing testimony parallelization
00:03:46.560 and how it works and how its performance changes
00:03:52.159 first uh wi-fi testing
00:03:57.519 cell testing benefits is here first in most cases we can learn test more
00:04:04.720 faster it means you can develop with clusters and speed in
00:04:10.400 tdd or vdd development
00:04:19.600 that is unique testing uh too long to learn you
00:04:25.280 need lives with any testing to
00:04:33.199 make target if you are committed to repository
00:04:40.240 faster testing test make developers more happy guess this is the example of development
00:04:46.560 cycles uh if you have slow tests first yeah you
00:04:52.160 implement some new feature or bug fixing
00:04:57.919 then you you learn test and check it process or it fails
00:05:03.199 but if test is slow you have to wait a long time this is also known as break time
00:05:13.039 and after that clone break use the red sentence means the test is failed
00:05:21.199 then you will fix the code and then we
00:05:26.560 learn the test as i said this this is very slow
00:05:32.240 you go to the second block this is also known as blade time as i
00:05:38.560 said
00:05:45.440 if you have a fast test first implement a new future
00:05:50.479 and learn the test and you can see a level with the first shorter time
00:05:56.000 and fix the impact is now perfect
00:06:01.199 you can now commit and push to the repository third best benefit is you can fix
00:06:07.840 failures with showtime periodization is the easiest way to
00:06:13.520 learn tests in showtime
00:06:18.720 i think some people have a question so here now uh in a little bit uh some changes uh
00:06:24.720 make make some uh some failures on another point uh so
00:06:30.400 long all tests at each commit is a recommended
00:06:36.720 this is a real-world example to make learning tests in showtime a quick but the most popular hp sharing class
00:06:44.080 in japan using ruby and lays first they used a power test that
00:06:50.400 parallel testing library for arabic the tests run faster than without
00:06:57.360 parallel tests but it still takes three minutes and half
00:07:02.960 is so long they try to land tests on remote machines which have a high performance
00:07:11.360 they have forced over to me to have very high performance they should use these
00:07:17.520 servers to learn this it's so high performance it's so expensive
00:07:23.840 they load the in-house library and it can learn by late tasks
00:07:29.440 first the library sends local code to remote server by other things to run test
00:07:35.120 then land tests and get output via ssh
00:07:41.759 okay i am not 19.2 times faster
00:07:47.120 parallelization is great thank you cookbot and uh tatino to allow
00:07:52.960 me to use this example
00:07:58.800 okay uh summary of benefits is soft status make development faster
00:08:04.720 and it means deployment faster so how we can learn the test in showtime
00:08:14.879 there are two ways to realize that and today i'll talk about parallel testing that is easiest way to learn
00:08:22.479 tests faster practicing
00:08:28.319 recently machines have multiple uh slits uh
00:08:33.360 anyone using uh anyone uh still using a single combustion in
00:08:38.640 in here maybe a thumbs okay no people
00:08:48.640 okay next and currently louis doesn't land by
00:08:54.399 because of some problem so learning about multiplayer through
00:09:02.399 tests using a slate class in ruby doesn't make sense
00:09:08.800 this is the installation of louis plate the bislet can
00:09:15.519 learn multiple slits at once
00:09:21.760 and this is the multiple process your machine's operating system may ground multiple processes at once and if
00:09:29.920 you have a multiple cores or a threat this is the difference between multiple
00:09:36.160 fluid processes and ruby level threats in this illustration test is
00:09:43.440 for process that testing and other is other any process like shell or
00:09:52.399 shell or browser or any next i'll talk about test uniform
00:09:59.920 relation that is the first vlogging by me this is not taking more long
00:10:05.760 time is left uh 30 minutes quantity
00:10:10.880 okay i think recently aspect is popular testing framework in movies so many people don't know testing it uh how many
00:10:18.560 people don't know a testing it is standard with ruby
00:10:24.959 do you know tesla test unit
00:10:30.560 oh it's not good i'm using uh r spec for my playback
00:10:36.560 private project i don't use testing it for my private
00:10:46.000 okay i like modifying luis g implementation uh i so i love some patches and land tests
00:10:53.200 but my machine is very slow so i wanted to make this test
00:11:00.079 test make target to test ruby has that name testo the more fast more and more
00:11:06.880 pressure i'll describe how the patch works and before but before this claim i have to
00:11:13.360 tell some tell some words to be used master is a process that
00:11:22.320 process the set first this process sends a instruction to uh
00:11:29.920 to work out master is a ruby process that started by
00:11:36.320 make targets was on and worker is a process that started by
00:11:43.600 a master process this process awaits instructions from
00:11:49.600 master process uh instruction is like a land this a test test
00:11:59.120 or some and that will get a long test
00:12:07.200 okay let's see how it works first user stats and master process by like make
00:12:14.639 test all next master process and the work processes
00:12:21.600 then master clauses send the given text name file names to workout sources
00:12:31.040 after work our leader test file work outlines uh test included in the file
00:12:38.959 and down the test result to master process finally master process sends another
00:12:46.000 test file name to remaining text files file names to workout process and start
00:12:53.120 from step 4 again repeat this
00:12:58.720 until the test file limit is leaving
00:13:04.079 okay this feature use standard input and output to communicate between workout
00:13:10.800 process and master process
00:13:15.839 aspects of parallel tests use
00:13:21.200 canal fork to start out on starter worker
00:13:27.040 but but i can't because test unit is used as ruby and
00:13:34.160 windows doesn't doesn't support folks i said
00:13:41.360 can i use folk but windows maintain a photo maintainer
00:13:46.639 with redundant if if this future doesn't work at windows
00:13:52.399 i'll reject it windows should not support it by
00:13:58.720 but i can't and this uh and this
00:14:04.880 feature parallelize each test file so you have to separate test files and test case
00:14:14.720 this is the illustration of learning steps first
00:14:20.240 user set the master process next master process starts work
00:14:27.199 with specified numbers of workers
00:14:32.399 using the iop open method this future use iop open methods
00:14:38.480 because windows doesn't have fork as i next said
00:14:43.839 workout process are sends test file names of tests to workout sources
00:14:51.760 workout process with test files from file system
00:14:58.160 and workout process lens test
00:15:03.680 finally workout return the result to master process and write next
00:15:09.040 file file name
00:15:14.160 and the master must send a remaining test file name to worker
00:15:21.760 and work out lands again and if there are no remaining test files
00:15:28.480 a master sends quit instruction to workers
00:15:34.160 and worker sends by instruction to master
00:15:39.600 and quits in default if there are any little contains
00:15:45.680 uh more than one fellows master process will utilize a failed
00:15:50.880 test on itself
00:15:56.800 there no pillars i i did i
00:16:02.480 implemented uh this little future because uh i can't uh
00:16:10.480 some tests uh still have issues to
00:16:15.600 uh to run in without parallelization
00:16:23.040 and then merge the layers and shows show to user via a standard output
00:16:30.720 okay this is all of mechanisms of this feature it works well but
00:16:36.639 in libya repository scientists don't consider about the population
00:16:43.279 this is not bad because of their parallelization in future
00:16:48.480 i fixed some tests in some bugs in tests by myself
00:16:55.360 okay i i found out issues in this data file in ruby
00:17:02.079 first test signal rv this is test around uh signal handling of ruby
00:17:09.120 this test the kernel forks method to check fork is implemented in this ruby because windows
00:17:16.799 doesn't have a fork method of their save when fogged processed are quitting
00:17:22.880 at exit hook at test unit kicked each time
00:17:27.919 it makes results outputting some strange so i either added that flag to testing
00:17:34.799 it at exit hook if the flag is true exist from the hook
00:17:39.840 as part of the hook and a
00:17:45.360 flag will be turned to turn to b2 uh if uh land uh
00:17:52.960 land more around one time okay next is the
00:17:59.280 test process lb this test modifies the standard input and output
00:18:04.799 this user used standard and input and output so i modified testonitomo worker
00:18:10.880 script named test unit parallel dot rb to duplicate
00:18:16.320 standard input and output before round test and workout process
00:18:23.039 and test http and https the test
00:18:29.919 net http library
00:18:36.640 use same port number and test i fix the port number to be
00:18:43.120 to be to be unique
00:18:50.880 i think the port number to be unique but fixed is committed by before i commit by uh committed by
00:18:58.240 another committer test csv uh use same
00:19:03.760 uh temporary directive name and they delete templates directly
00:19:09.120 in teardown phase so under test there is a template directly
00:19:15.200 so it will be a like this
00:19:23.200 no such file or directory error is recorded in result
00:19:29.760 so i'll rename the template directly names in the tests to be unique each test in
00:19:35.679 test csv okay this is the last issue
00:19:41.840 test of lobby games some tests of movie games forgot lighting
00:19:47.200 dependent files dependence dependency is a file should be required before running tests
00:19:55.280 if there are test a and test b script and the library name there are four dots
00:20:01.520 are b and test a and test b depends on food rb
00:20:07.280 test a includes regulating foo but test b doesn't
00:20:15.919 when learned this test without the parallelization it'll be no errors
00:20:22.240 because who is already required in same ruby process
00:20:27.840 but when using a parallelization it will be an error
00:20:33.280 because who is loaded um one will be processed but it is not required in other one to
00:20:40.000 be processed i'm fixing iui hand and i've found only in lubby
00:20:47.200 games this currently i don't know why but this case can be found in
00:20:54.080 any test but i found this currently founded in louis james tests only
00:21:00.880 okay to uh this is the message is uh to louis james developers
00:21:06.240 please match uh revision three three two three two uh in ruby into your gate repository and
00:21:13.600 slowly if the touch has always merged if you alignment please close the ticket thank you
00:21:21.360 okay i tell you a message to the beginning table will be james developers ah
00:21:26.400 in here anyone will begin uh developer of the museum louisiana
00:21:33.039 okay no one uh someone uh to tell this uh
00:21:38.159 instead of me uh how's the performance uh um the
00:21:44.720 message was one of important things in my talk but um no regimes developed in here seems um
00:21:53.840 okay how's uh performance changed without power transition and without
00:21:58.880 parallelization akin to mulatto also known as mlckn
00:22:05.120 molokem he measured a playful time of both and created a performance graph he
00:22:12.240 will talk at the last session of loom2 about the release number system
00:22:17.440 around the road i think it's one of the important thing if you are interested in
00:22:22.799 lubi's future if you are in uh i i i think after like
00:22:28.240 this slide uh uh i think you are not here if uh you are
00:22:34.400 interested in living future because louise future uh talk is uh
00:22:39.919 learning current learning on loom2 okay this is the environment that
00:22:46.720 measured performance shown here
00:22:51.919 and this is a graph band name single is level of wizard parallelization and
00:22:58.400 looks uh like amazingly faster to
00:23:04.720 this is a little as shown as the exact number test test take 121.4.49
00:23:14.159 seconds without polarization but it will be
00:23:19.880 43.41 seconds with enabling
00:23:25.440 polarization amazingly fast 2.9 and
00:23:32.640 79 times more faster than without polarization
00:23:39.600 everything we passed okay last is how to use this but i have to
00:23:49.120 plan some demonstration before i describe it
00:24:13.279 okay
00:24:30.000 this is our tested learning
00:24:40.320 oops okay
00:24:48.240 is there any
00:24:54.080 oops
00:25:05.520 oops uh i
00:25:11.520 i want to show uh test is parallelized
00:25:17.440 but i can see the oh
00:25:23.840 this uh in this uh lines uh
00:25:30.080 uh cj cookie test and uh benchmark uh vm test
00:25:36.080 is learning power reality because uh it's shown uh
00:25:46.799 output from workers
00:25:52.240 this is uh if not using a polarization this will be
00:25:58.159 one test case this class name in one on is one atoms
00:26:06.480 and this is one this is the more one
00:26:11.600 benefit of testing prioritization uh
00:26:18.960 with in result of parallelization as shown here starting is very slow because uh
00:26:26.640 test unit with without preparation is uh load first require first
00:26:33.679 require first all of test files so if uh
00:26:40.240 stats slow by
00:26:45.440 visualization because liquiang is slow
00:26:51.039 in this in with uh in in uh enabling uh parallelization uh it will
00:26:57.760 the uh test will be start fast uh because uh requiring requiring uh test
00:27:03.679 files will be defiled okay let's back to uh how to use it
00:27:12.720 okay uh how to use class super test case to
00:27:20.640 multiple file and next we write script to learn multiple text file is using a test unit
00:27:28.960 outliner uh the scripts good example is ruby's test lana
00:27:34.159 dot i'll be in louisville's repository next lanza's clipped with argument uh
00:27:41.679 high jr j option and
00:27:46.840 visa number of number of workers learning the script with
00:27:54.399 help provides more information
00:28:00.399 and the patch made me a committer of ruby i was very happy then
00:28:09.120 okay that's all uh before i have um
00:28:15.120 i finished the flashlight i have one announcement lui 1.9.3
00:28:21.840 uh release candidate one has been released you can use parallelization by installing this and let's try it and
00:28:29.120 tell us if you found a bug more about it is available at
00:28:34.240 www.libbyhathonline.org okay
00:28:39.919 any question over there
00:28:47.440 you've made a really compelling uh case for parallel testing how do we get developers to actually do that
00:28:53.760 uh without like forcing them
00:29:10.799 uh
00:29:24.159 many developers don't if you don't use this feature but some computers are
00:29:31.200 using the future including me
00:29:37.200 some some committees i think they are thinking
00:29:42.399 this future is still unstable but it is working stable
00:29:48.799 please use it okay
00:29:54.080 okay uh currently this only works with test unit do you think it will be possible to make work with our spec too
00:30:02.320 i think have already uh generated plugging uh i'm not thinking quantity
00:30:11.760 any other nope
00:30:18.720 okay
00:30:34.559 this is a gift from japan the final will be teacher
00:30:40.559 if you wanted this uh please come here and take it
00:30:48.559 okay attention available in here
00:31:38.240 you
S
Shota
1 talk
F
Fukumori
1 talk
Explore all talks recorded at RubyConf 2011
+55