Portman CLI 1.0 - Better API testing

API testing libraries like Portman make it easy to ensure your APIs are reliable while keeping your documentation in sync with the API implementation. Portman CLI takes care of the OpenAPI Spec to Postman conversion while injecting contract & variation tests with a minimum of configuration. It includes options to customize Postman requests & variables with a wide range of settings to assign & overwrite variables.

Portman leverages your static OpenAPI specs, with all its defined API request/response properties, to power your Postman collection and test suite.

Since we released the beta, we've also added the ability to orchestrate and run integration tests against your API via Postman & Newman!

With Portman CLI 1.x, you can:

• Convert an OpenAPI document to a Postman collection
• Upload the Postman collection to your Postman app and workspace
• Integrate API testing into your CI/CD pipeline
• Test the Postman collection via Newman
• Extend the Postman collections with collection variables and much more.

Give it a try through NPM

npm add --save-dev @apideck/portman@latest

From PR to Portman CLI

It started with the need to transform an OpenAPI document to be used as a Postman collection. Portman is the result of the "spec-driven" approach and the desire to automate that comes with it.

This meant, spending less time maintaining a Postman collection while keeping it in sync with the OpenAPI changes since it was generated after each change made to the OpenAPI document. To transform the OpenAPI doc to a Postman collection, we used the handy openapi-to-postman package from @getpostman themselves.

The next step was to generate some very basic Postman tests, sourced by OpenAPI specification itself. We created a PR on the openapi-to-postman package, that started with a basic HTTP status code check, inserted during the transformation. Gradually, more standard Postman tests like response content-type, execution time etc were added to facilitate more automation in Postman.

As it became clear we needed to modify some of the default examples values from OpenAPI, the PR grew, more test generation functions were added, but it started to feel inappropriate to add them to openapi-to-postman package since the package had a specific task - converting OpenAPI to Postman - and that test generation did not fully belong in there.

Apideck was building a CLI package to manipulate Postman collections to facilitate automation, with methods to replace keys/values, insert variables from .ENV, inject pre-requests scripts in Postman.

As both initiatives had similar goals, they were combined into what is now known as 👨🏽‍🚀 Portman.

Goal

The goal of Portman is to drive API test automation by 'porting' a static OpenAPI document to a dynamic Postman collection that includes a powerful testing suite with variable requests, bodies, and more. All while being easy to configure & ready to use.

With Portman, you can:

• Convert an OpenAPI document to a Postman collection
• Upload the Postman collection to your Postman app
• Test the Postman collection via Newman
• Manage everything in the config file for easy local or CI/CD usage
• Extend the Postman collection with capabilities
  • Assign collection variables
  • Inject Postman Contract Tests
  • Inject Postman Variation Tests (release 1.1.0)
  • Inject Postman Integration Tests (release 1.2.0)
  • Inject Postman pre-request scripts on a collection or operation level
  • Modify Postman requests and use responses to assign to variables

The Power of Open Source & OpenAPI

Portman CLI is built on top of powerful open-source tools & packages:

• OpenApiSpec: The OpenAPI Specification
• openapi-to-postman: Package to convert OpenAPI to Postman
• Postman: collaboration platform for API development
• Newman: command line Postman collection runner

Although Portman seems very similar to the openapi-to-postman package, there is an important distinction. While openapi-to-postman focuses on transforming the OpenAPI document to a Postman collection, Portman strives to automate the functionality to generate tests and modify the Postman collection based on OpenAPI.

Portman in Action

Let's show a simple example of converting OpenAPI to Postman with a range of Portman contract tests, automatically generated, without any complex configuration or manually written tests.

portman -u https://specs.apideck.com/crm.yml -c portman-config.crm.json -o crm.postman.json

The Portman settings (in JSON format) can consist of multiple parts.

In this example, we focus on the contractTests section and settings.

{
  "version": 1.0,
  "tests": {
    "contractTests": [
      {
        "openApiOperation": "*::/crm/*",
        "statusSuccess": {
          "enabled": true
        }
      },
      {
        "openApiOperation": "*::/crm/*",
        "responseTime": {
          "enabled": true,
          "maxMs": 300
        }
      },
      {
        "openApiOperation": "*::/crm/*",
        "contentType": {
          "enabled": true
        }
      },
      {
        "openApiOperation": "*::/crm/*",
        "jsonBody": {
          "enabled": true
        }
      },
      {
        "openApiOperation": "*::/crm/*",
        "schemaValidation": {
          "enabled": true
        }
      },
      {
        "openApiOperation": "*::/crm/*",
        "headersPresent": {
          "enabled": true
        }
      }
    ]
  }
}

The Portman configuration instructs Portman to generate tests in Postman for all OpenAPI operations that match the combination of the OpenAPI methods (GET, POST, PUT, DELETE, ...) & paths /crm/*.

Explanation of the test settings:

• statusSuccess: Adds the test if the response of the Postman request returns a 2xx
• responseTime : Adds the test to verify if the response of the Postman request is returned within a number of ms.
• contentType : Adds the test if the response header is matching the expected content type defined in the OpenAPI spec.
• jsonBody : Adds the test if the response body is matching the expected content type defined in the OpenAPI spec.
• schemaValidation : Adds the test if the response body is matching the JSON schema defined in the OpenAPI spec. The JSON schema is inserted inline in the Postman test.
• headersPresent : Adds the test to verify if the Postman response header has the header names present as they are defined in the OpenAPI spec.

The result will be that the initial OpenAPI file, with all request and response details is used to generate the specific tests. Per Postman request, you can review the specific tests in the "Tests" tab in the Postman application.

While this is a simple example to introduce some of the functionality of Portman, it is equally as easy for Portman to generate a suite of Variation Tests - to test your API's error handling - as well as create an Integration Test suite - to test application state and data persistence.

Detailed documentation and further examples can be found on Github and NPM.

In the following weeks, we will be releasing blog posts covering the different testing use cases and strategies. We're looking forward to hearing your feedback!