{ "version": "https://jsonfeed.org/version/1.1", "title": "Joe Smith's Blog", "language": "en", "home_page_url": "https://bjoli.com/", "feed_url": "https://bjoli.com/feed/feed.json", "description": "I'm a software engineer stoked about Open Source Software & scalable infrastructure.", "authors": [ { "name": "Joe Smith", "url": "mailto:yasumoto7@gmail.com" } ], "items": [ { "id": "https://bjoli.com/posts/pathfinder/", "url": "https://bjoli.com/posts/pathfinder/", "title": "Pathfinder Society", "content_html": "
\n

I love to step away from the computer to play the tabletop roleplaying game,\n Pathfinder.

\n

Bay Area Pathfinder Society

\n

I'm a Venture Lieutenant for San Francisco,\n which means I help organize events and conventions, in addition to\n running our Facebook page.\n If you're in the San Francisco Bay Area, there are a lot of great games hosted by the\n Bay Area Pathfinder Society! In particular,\n Endgame in Oakland hosts events on Monday\n evenings, and Gamescape in San Francisco\n hosts a Saturday game day each month.

\n

Publications

\n

Zenith Games recently published\n The Colossal Creatures Bestiary\n which contains a massive Raccoon-like creature I designed called a Procyon! It was a great\n experience to work with that group, and it's available\n on the Paizo webstore\n if you'd like to take a look!

\n

The fine folks behind Wayfinder #13\n also published my description of two characters based in the city of Caliphas! Spanning two\n pages and 1500 words, you can read about both Captain Almira Perine\n and Captain Liald Gaspair! The character art is by the\n amazing Bob Greyvenstein.\n

\n
\n", "date_published": "2016-10-24T00:00:00Z" } , { "id": "https://bjoli.com/posts/axes_of_testing/", "url": "https://bjoli.com/posts/axes_of_testing/", "title": "Axes of Testing", "content_html": "
\n

Axes of Testing

\n

I've tried many different tacks when explaining the benefits of unit testing to other\n developers (and proving to myself that they are worth it). Tests are not a silver bullet,\n they are just another tool in your war chest, and certainly have their limitations. What\n follows is the best breakdown I've come up with to describe the various types of tests and\n when to lean on them. I presented this at a Twitter-internal Python tech talk, and many\n teammates nudged me to publish it externally. I hope it will allow others to understand how\n to approach adding different types of tests for your codebase.

\n

The Spectrum of Testing

\n

Big and Small, Real and Fake

\n

Initially, I started using a single axis to try to delineate between \"small\" unit tests and\n \"widely-scoped\" integration tests. This wasn't effective, as I got lost when trying to figure\n out how to deal with moving away from real dependencies (sending prod traffic to other\n services) vs. making sure I replaced those interactions with mocks or fakes so I could still\n run my tests on an airplane.

\n

This helped me realize that I was actually staring at (at least) a two dimensional spectrum,\n and thus the axes were born.

\n

Axes

\n

\"bare

\n

Horizontal

\n

The first, horizontal axis covers whether or not you're sending traffic to live\n production, or just mocked out services. If you run curl http://twitter.com,\n then you're hitting Twitter's website. However, if you change\n your /etc/hosts file to instead point that hostname to your localhost where\n you're running a web server, then you've taken a step away from \"production\" and toward a\n faked out twitter.com.\n

Vertical

\n

The second, vertical axis is used to show the \"size\" of a test— how much of the system is\n exercised when executed? Using that curl example above, that is a test that\n sweeps the gamut of infrastructure. It ensures that the curl binary is\n installed on your system, your machine is connected to the Internet (with DNS, etc.), but\n also that Twitter's front-end infrastructure is serving correctly. This requires all of\n those pieces to work, and an error in one component will cause a failure for the entire\n test.

\n

At the low end, you have a test which is akin to a shell script called ./curl\n which just ensures you pass in http://twitter.com as the only argument. This\n isolates your execution from all other variables than just the way you're calling\n curl, but the downside is you're not really ensuring that you are using things\n correctly. If, for instance, curl gets updated to change the way it takes a\n URI for an argument, then you won't notice by running your test script, since it's not\n actually invoking curl at all. Also, if (heaven forbid..)\n twitter.com goes down or moves locations, you won't notice\n that either.

\n

Extremes

\n

Let's take a look at what would happen if we just followed some of these options to the max

\n

\"extremes\"

\n

Brittle

\n

If you are testing the whole end-to-end pipeline of a task, then a small break in any piece\n will grind your entire test to breakage. Large, broken test suites are tough to debug\n because you won't be able to tell which component is causing your failure without looking\n at the error message in detail.

\n

Dependent Services

\n

If you have a service which has zero mocks, and fully depends on all of production\n responding and working at all times, then it will break anytime there is an issue\n with any small piece. Many downstreams in a test can be difficult to keep running\n frequently and reliably without a ton of work to ensure they are all stable.

\n

Interaction Validation

\n

If you have very small-scoped tests, then you may miss huge interaction problems, and may\n not realize that you are sending the wrong format of data between various services, or\n perhaps are not rate-limiting the total number of requests (or any of an infinite number)\n of interactions.

\n

No Confidence

\n

If everything is mocked out, then you're spending all your effort on re-building\n infrastructure and pieces, without gaining any benefits of testing the actual systems you\n care about!

\n

Worst of All Worlds

\n

We can get even worse by blindly combining some of these ideas as well!

\n

\"bad

\n

Testing In Production

\n

If you want to ensure you're causing problems for your users, throw some additional monkey\n wrenches into the works. This isn't the same as Netflix's Chaos Monkey; I'm talking about\n throwing brand-new, un-vetted code into production so your customers detect the issues at\n the same time you do. Do not send huge request loads onto your brand new systems without\n even validating they pass health checks first!

\n

Submit Queue

\n

If you have many small tests that all hit various portions of live infrastructure, you will\n assuredly find lots of broken pieces over time. This will be a constant drag on\n productivity, as engineers will never quite know if the production service is having an\n issue or the code they just wrote is broken without investigation.

\n

API Changes

\n

If you have many, small tests and mock everything out, then you won't actually see what\n big interface changes happen between modules or services. This will lead you to writing a\n lot of tests that don't verify anything, and not prevent you from changing the wrong thing\n which breaks a big service and takes the site down.

\n

Maintaining Mock Services

\n

If you want to have an equally-staffed Testing Team who shadows your \"primary\" team of\n developers, then you can likely manage to build a \"stack in a box\" and build the massive\n fleet required to mock out all of your services and keep up with changes. Otherwise,\n probably not worth the time investment.

\n

Unit Tests and Integration Tests

\n

I believe that by taking a middle path between these extremes, we can come up with two groups\n of tests that will make developers lives much better.

\n

\"compromises\"

\n

Terminology

\n

Unit Tests

\n

These are tests that should be scoped relatively small, and rely on mocking any external\n service that makes a network call or IO of any kind. This allows these tests to be run\n after every commit (or after every change!) and ensure that the feedback loop is very\n tight. Even running a linter would be a small start, but code you write to validate your\n production code does what it should would be best.

\n

Integration Tests

\n

These are the larger, less frequently run tests. This would be part of a frequent release\n process, something that wasn't so frequent that it was expected to be run by developers\n multiple times per day, but not so infrequently that things change out from underneath it\n too frequently. This is something that could actually spin up dev instances of whatever\n services are necessary, and run through a battery of expected user behavior.

\n

Conclusion

\n

Between small and fast \"unit tests\" and larger, more comprehensive \"integration tests\"\n I have found that it enables teams to move more quickly and with confidence, even if those\n engineers are just dropping in with a quick bug fix. Comments and suggestions appreciated-\n feel free to mention me on Twitter!\n

\n", "date_published": "2016-07-21T00:00:00Z" } ] }