Pragmatic Testing With Postman And Newman
While large corporations with dedicated test engineers may value feature rich test frameworks, they’re not for everyone. Is there a simpler way to test web services and get more bang for your buck?
Testing Foundations
Let me start by saying I am not a test engineer, nor a QA. I’m not a specialist in testing at all. As a developer though, I value having testable and well tested applications.
When working in an development team, it is normal — recommended — to have a mixture of both developers and testers. This separation allows different sets of eyes on the feature under development, while allowing a degree of specialism.
Having a different set of eyes on something helps to overcome the developers optimistic bias. Developers can often concentrate on the happy path, particularly when they have limited experience; getting something working well enough to meet the requirements is their primary goal. With another, more sceptical, perspective there can be a focus on the non-happy path too.
Specialism also brings value. Different folks have different skills and approaches to what they do. Developers often revel in the shape and details of the code, while testers are more concerned with the practicalities of whether it works and is reliable.
Pragmatic Testing
While it would be great to always have many skilled professionals, fettling and refining to a high degree, this comes with a cost. A substantial cost. Can testing be done well on a tighter budget?
I’ll start by saying there is no replacement for a second pair of eyes. With experience though, a developer can certainly consider more perspectives. The cognitive load of writing the code gets lower with practice and gaining insight into common non-happy paths makes them more avoidable.
What is unavoidable is the need to write automated tests to confirm the application is behaving as desired. At the beginning this may not feel necessary, but a few weeks into development, it will become obvious — requirements are changed or tweaked, refactoring becomes necessary as the design evolves, etc.
So, how can these dual challenges — testing your own code and creating automated tests — be met?
I’ll start by asserting that unit testing is being done in concert with development. Test Driven Development (TDD) is not new, but it becomes essential as applications grow. No matter how experienced a developer is, they will need to refactor sooner or later. Sometimes, this is within days, other times it is within weeks. You will need them to be present whenever such changes are required, if you are to retain confidence in the software.
This leaves acceptance or regression (service) testing as the next big slices of the Test Pyramid.
Black Box Testing
For testing web services, there are many tools for black box testing. These will help to test against an acceptance criteria, then subsequently check for any regressions.
There are dedicated test frameworks, such as TestNg on Java. There are many libraries for testing RESTful endpoints, such as RestAssured. There are many ways of using either, from XML or CSV test specifications, through to code. There are many hybrids and custom libraries. Each of these requires a degree of configuration and development.
For black box testing, I’m going to go out on a limb and say much of this is not necessary, especially for smaller applications/services. That isn’t to say that these solutions aren’t feature rich and can be incredibly powerful. It just means that they provide more than you need for pragmatic testing.
Enter Postman
I’ve worked on many projects over the last few years, in companies of all shapes and sizes. One consistency has been the use of Postman for sending requests to web services. Sometimes these are informal, throw away, uses. Other times, there have been large, well organised Postman Collections. Regardless, it has become a bit of a Swiss Army Knife for web service development and testing.
What seems to be less common is using these Postman Collections as a simple test suite.
Feel free to mock (excuse the pun!), but when we have a bunch of Postman requests, it would seem a shame to duplicate them all to form a test suite. You then have at least twice the effort to create them and maintain them.
Postman Collections can be ordered and each request can have a number of tests conditions, expressed using JavaScript. These can all be defined within the Postman client, with plenty of templating options and help documents.
Once you have your Collection defined, you can run it from the Postman client. This will fire off each request, test the response and produce a graphical report, detailing what passed or failed.
Enter Newman
Better still, you can export a Collection and run it using the Newman command, from a command line interface (CLI). This makes it trivial to run the suite repeatedly — there is even a switch to loop many times for soak testing.
Having a CLI interface brings numerous other advantages. Being able to automate the execution of the suite is a huge benefit and dropping the command into your CI/CD flow delivers this. Using the same Postman requests that were used for manual testing. Less duplication, means less effort wasted.
As a developer, I create Postman requests to confirm my code is working. Wearing my tester hat, I can then complement these tests easily with non-happy path alternatives. I can maintain these with little effort and run the whole suite quickly when I need to. This saves me time and helps to improve my test coverage.
Can it do everything? No, there is no option for white box testing. I can’t check database state and so forth, so I’m restricted to testing the response. I can’t add data fixtures unless I can add them through a service call either. Is this sufficient for a basic test suite? In my experience, yes it is.
Shouldn’t I just be coding it? No… at least not when creating a pragmatic test suite. With Postman Collections, I can hand them to a client, a business analyst, a product owner, a non-Java developer or a test engineer. They are easy to understand, edit and maintain without requiring skilled development experience.
While I could code the tests, it can bring the wider team more value if I don’t.
Isn’t that what being pragmatic is all about?
