PACT Conformance Testing Checklist

1. Introduction

Conformance testing is a necessary process to ensure interoperability between independent implementations of the PACT Technical Specifications.

This guidance document supports manual conformance testing processes by defining required and conditional test cases. Advisement: This document does not replace the technical specifications. It is possible that there are deviations between this document and the technical specifications. These cases are unintended. This document also does not superseed or otherwise have normative value over the technical specifications.

2. Terminology

Required Test Case

Test case relating to MANDATORY functionality

Conditional Test Case

Test cases relating to OPTIONAL functionality.

Host System

see PACT Tech Specs V2.2 § host-system

testing party

The party witnessing and perforing the tests. Typically, the tester is a data recipient

Target Host System

The host system under test.

3. Scope

This document defines test cases to be used for testing the interoperability of independent implementations of the PACT Technical Specifications.

A test case contains instructions to be executed by a testing party against a target host system so that the testing party can compare the behavior of the target host system with the expected behavior according to the technical specifications.

Test cases for the following functionality are defined in § 5 Required Test Cases:

• All actions are offered under the https method only (i.e., http requests are not processed)

• The endpoint /auth/token is exposed (even if a custom AuthEndpoint is used for authentication)

• The host system supports the following actions:

  • Action Authenticate

  • Action ListFootprints, including pagination and support for the limit parameter

  • Action GetFootprint

• If the host system does not support Action Events, it responds to authenticated Action Events calls with an error response with code NotImplemented.

There are also test cases for conditional or optional functionality defined in § 6 Conditional Test Cases:

• filtering of the ListFootprints action (see PACT Tech Specs V2.2 § api-action-list-filtering)

• OpenID Connect-based authentication flow (see PACT Tech Specs V2.2 § api-auth)

• HTTP Action Events (see PACT Tech Specs V2.2 § api-action-events)

Note: Support for Action Events will become mandatory in v2.2 of the Technical Specifications.

4. Testing Preparation

Prior to all tests, the owner of the target host system must provide the testing party with

1. credentials that the testing party for successful authentication (i.e., a client_id and a client_secret)

2. the authentication URL to use (see PACT Tech Specs V2.2 § authhostname)

3. the base URL to the different HTTP Actions defined in the PACT Technical Specifications (see PACT Tech Specs V2.2 § hostname)

5. Required Test Cases

5.1. Test Case 001: Authentication against default endpoint

Tests the target host system's ability to authenticate a data recipient through the Action Authenticate endpoint offered through the default path /auth/token:

5.1.1. Request

An authentication POST request must be sent to the auth/token endpoint of the test target host system with correct credentials the syntax specified in PACT Tech Specs V2.2 § api-action-auth-request (the credentials need not be correct).

From v2.1 onwards, host systems can also use a custom AuthEndpoint for authentication, specified in an OpenId Provider Configuration Document. This is tested through § 6.1 Test Case 012: OpenId Connect-based Authentication Flow.

5.1.2. Expected Response

Depending on whether authentication through /auth/token is supported, the test target host system must respond with either

• 200 OK and a JSON body containing the access token, as specified in PACT Tech Specs V2.2 § api-action-auth-response, or

• 400 Bad Request and a JSON body containing an error, as specified in PACT Tech Specs V2.2 § api-action-auth-response. In this case, the testing party must execute the test case § 6.1 Test Case 012: OpenId Connect-based Authentication Flow.

5.2. Test Case 002: Authentication with invalid credentials against default endpoint

Tests the target host system’s ability to reject an authentication request with invalid credentials through the default endpoint /auth/token.

5.2.1. Request

A test case similar to § 5.1 Test Case 001: Authentication against default endpoint but performed with incorrect credentials (i.e. the client id and/or client secret are unknown to the target host system).

5.2.2. Expected Response

The target host system responds with 400 Bad Request and a JSON body containing the error "invalid_client", as specified in PACT Tech Specs V2.2 § api-action-auth-response.

5.3. Test Case 003: Get All Footprints

Tests the target host system’s ability to list all available PCFs.

5.3.1. Request

A ListFootPrints GET request must be sent to the /2/footprints endpoint of the test target host system with a valid access token and the syntax specified in PACT Tech Specs V2.2 § api-action-list-request.

No additional request parameters, such as limit or filter, must be defined.

5.3.2. Expected Response

The test target host system must respond with 200 OK and a JSON body containing the list of all available PCFs, as specified in PACT Tech Specs V2.2 § api-action-list-response.

5.4. Test Case 004: Get Limited List of Footprints

Tests the target host system’s ability to list a limited number of PCFs when the limit parameter is set by the data recipient.

5.4.1. Request

A ListFootPrints GET request must be sent to the /2/footprints endpoint of the test target host system with the limit parameter, a valid access token and the syntax specified in PACT Tech Specs V2.2 § api-action-list-request.

5.4.2. Expected Response

The test target host system must respond with

1. either an HTTP status code 200 "OK" with a response body containing a list of PCFs with a length equal to or smaller than the limit set in the request, as specified in PACT Tech Specs V2.2 § api-action-list-response. Unless the total number of available PCFs is equal to or smaller than the limit set in the request, the test target host system must return a Link header

2. or an HTTP status code 202 and an empty body

Note: For testing purposes it is recommended to set the limit to a small number (e.g., 2) to ensure that pagination is tested.

5.5. Test Case 005: Pagination link implementation of Action ListFootprints

Note: This test pressuposes the completion of § 5.4 Test Case 004: Get Limited List of Footprints and uses the link returned in the header. If § 5.4 Test Case 004: Get Limited List of Footprints fails, this test can be skipped.

Tests the target host system’s ability to return PCFs when the same pagination link, returned through the link header, is called multiple times.

5.5.1. Request

The testing party calls the ListFootprints action (PACT Tech Specs V2.2 § api-action-list-request) with a valid access token such that a link header is returned by the target host system. The testing party selects a pagination link from a link header at random to perform the tests against. The testing party then calls the pagination link 2 or more times.

This test must conclude within 180 seconds after the pagination link was retrieved originally.

5.5.2. Expected Response

The test target host system must respond with either 200 OK or 202 Accepted and a JSON body containing PCFs. The contents of the response bodies should be the same across all calls to the pagination link.

5.6. Test Case 006: Attempt ListFootprints with Expired Token

Tests the target host system’s ability to reject a ListFootprints request with an expired access token with the correct error response.

5.6.1. Request

A ListFootprints GET request must be sent to the /2/footprints endpoint of the test target host system with an expired access token and the syntax specified in PACT Tech Specs V2.2 § api-action-list-request.

5.6.2. Expected Response

The test target host system must respond with a 401 Unnauthorized and a JSON body that should contain the error response TokenExpired, as specified in PACT Tech Specs V2.2 § api-error-responses.

Note: Since the access token is expired, re-authentication should in principle solve the issue. By returning the HTTP error code 401 (instead of, e.g., 403), the host system signals that re-authentication should be attempted.

5.7. Test Case 007: Attempt ListFootPrints with Invalid Token

Tests the target host system’s ability to reject a ListFootprints request with an invalid access token with the correct error response.

5.7.1. Request

A ListFootprints GET request must be sent to the /2/footprints endpoint of the test target host system with an invalid access token and the syntax specified in PACT Tech Specs V2.2 § api-action-list-request.

5.7.2. Expected Response

The test target host system must respond with a 403 Forbidden and a JSON body containing the error response AccessDenied, as specified in PACT Tech Specs V2.2 § api-error-responses.

Note: Since the access token is invalid, re-authentication cannot solve the issue. By returning the HTTP error code 403 (instead of, e.g., 401), the host system signals that there is no gain in re-authenticating.

5.8. Test Case 008: Get Footprint

Tests the target host system’s ability to return a PCF with a specific pfId. This pfId must correspond to one of the PCFs returned by the ListFootprints action.

5.8.1. Request

A GetFootprint GET request must be sent to the /2/footprints/{GetPfId} endpoint of the test target host system with a valid access token and the syntax specified in PACT Tech Specs V2.2 § api-action-get-request.

5.8.2. Expected Response

The test target host system must respond with 200 OK and a JSON body containing the PCF with the requested pfId, as specified in PACT Tech Specs V2.2 § api-action-get-response.

5.9. Test Case 009: Attempt GetFootprint with Expired Token

Tests the target host system’s ability to reject a GetFootprint request with an expired access token with the correct error response.

5.9.1. Request

A GetFootprint GET request must be sent to the /2/footprints/{GetPfId} endpoint of the test target host system with an expired access token and the syntax specified in PACT Tech Specs V2.2 § api-action-get-request.

5.9.2. Expected Response

The test target host system must respond with a 401 Unnauthorized and a JSON body that should contain the error response TokenExpired, as specified in PACT Tech Specs V2.2 § api-error-responses.

Note: Since the access token is expired, re-authentication should in principle solve the issue. By returning the HTTP error code 401 (instead of, e.g., 403), the host system signals that re-authentication should be attempted.

5.10. Test Case 010: Attempt GetFootprint with Invalid Token

5.10.1. Request

A GetFootprint GET request must be sent to the /2/footprints/{GetPfId} endpoint of the test target host system with an invalid access token and the syntax specified in PACT Tech Specs V2.2 § api-action-get-request.

5.10.2. Expected Response

The test target host system must respond with a 403 Forbidden and a JSON body containing the error response AccessDenied, as specified in PACT Tech Specs V2.2 § api-error-responses.

Note: Since the access token is invalid, re-authentication cannot solve the issue. By returning the HTTP error code 403 (instead of, e.g., 401), the host system signals that there is no gain in re-authenticating.

5.11. Test Case 011: Attempt GetFootprint with Non-Existent PfId

Tests the target host system’s ability to reject a GetFootprint request with a non-existent pfId with the correct error response.

5.11.1. Request

A GetFootprint GET request must be sent to the /2/footprints/{GetPfId} endpoint of the test target host system, where {GetPfId} is a non-existent pfId, with a valid access token and the syntax specified in PACT Tech Specs V2.2 § api-action-get-request.

5.11.2. Expected Response

The test target host system should respond with a 404 Not Found and a JSON body containing the error code NoSuchFootprint, as specified in PACT Tech Specs V2.2 § api-error-responses.

6. Conditional Test Cases

6.1. Test Case 012: OpenId Connect-based Authentication Flow

Tests target host system's ability to authenticate a requesting data recipient through a custom AuthEndpoint.

6.1.1. Condition

The target host system supports the OpenId connect-based authentication flow (see PACT Tech Specs V2.2 § api-auth).

6.1.2. Request

Following the OpenId Connect-based authentication flow, the testing party retrieves the OpenId Provider Configuration Document.

The testing party then authenticates through the AuthEndpoint referenced in the Configuration Document as specified PACT Tech Specs V2.2 § api-action-auth-request.

6.1.3. Expected Response

1. The target host system returns a valid OpenId Provider Configuration Document

2. The test target host system responds with 200 OK and a JSON body containing the access token, as specified in PACT Tech Specs V2.2 § api-action-auth-response upon the testing party authenticating through the token endpoint referenced in the Configuration Document.

6.2. Test Case 013: OpenId connect-based authentication flow with incorrect credentials

6.2.1. Condition

The target host system supports the OpenId connect-based authentication flow (see PACT Tech Specs V2.2 § api-auth).

6.2.2. Request

The testing party performs the same flow as in § 6.1.2 Request but with incorrect credentials.

6.2.3. Expected Response

1. The target host system returns a valid OpenId Provider Configuration Document

2. The target host system respons with a 400 Bad Request and a JSON body containing the error "invalid_client", as specified in PACT Tech Specs V2.2 § api-action-auth-response.

6.3. Test Case 014: Attempt Authentication through HTTP (non-HTTPS)

According to PACT Tech Specs V2.2 § api-requirements, a host system must offer its actions under https method only.

6.3.1. Request

An http-only equivalent of the test target host system AuthEndpoint (be it /auth/token or a custom endpoint) must be generated, replacing "https://" by "http://".

An authentication POST request must be sent to the generated http endpoint with the the syntax specified in PACT Tech Specs V2.2 § api-action-auth-request (the credentials need not be correct).

6.3.2. Expected Response

The target host system either refuses to process the request (for instance the HTTP port 80 is not open) or responds with an HTTP error response code.

6.4. Test Case 015: Attempt ListFootprints through HTTP (non-HTTPS)

According to PACT Tech Specs V2.2 § api-requirements, a host system must offer its actions under https method only.

6.4.1. Request

An http-only equivalent of the test target host system ListFootprints endpoint must be generated, replacing "https://" by "http://".

A ListFootprints GET request must be sent to the generated http endpoint with the the syntax specified in PACT Tech Specs V2.2 § api-action-list-request (the access token need not be valid).

6.4.2. Expected Response

The target host system either refuses to process the request (for instance the HTTP port 80 is not open) or responds with an HTTP error response code.

6.5. Test Case 016: Get Filtered List of Footprints

Tests the filtering implementation of a target host system's ListFootprints action (see PACT Tech Specs V2.2 § api-action-list-filtering).

6.5.1. Condition

The target host system supports filtering.

6.5.2. Request

A ListFootPrints GET request must be sent to the /2/footprints endpoint of the test target host system with the filter parameter, a valid access token and the syntax specified in PACT Tech Specs V2.2 § api-action-list-request.

6.5.3. Expected Response

The test target host system should respond with 200 OK and a JSON body containing a list of PCFs matching the filtering criteria.

6.6. Test Case 017: Attempt GetFootprint through HTTP (non-HTTPS)

According to PACT Tech Specs V2.2 § api-requirements, a host system must offer its actions under https method only.

Therefore, it is our understanding that host systems must not expose any action-related endpoints through http (non-https).

6.6.1. Request

An http-only equivalent of the test target host system GetFootprint endpoint must be generated, replacing "https://" by "http://".

A GetFootprint GET request must be sent to the generated http endpoint with the the syntax specified in PACT Tech Specs V2.2 § api-action-get-request (the GetPfId need not exist).

6.6.2. Expected Response

The target host system either refuses to process the request (for instance the HTTP port 80 is not open) or responds with an HTTP error response code.

6.7. Test Case 018: Receive Notification of PCF Update

This test case will become mandatory test case with the release of version 2.2 of the Technical Specifications.

Tests the target host system’s ability to be notified of a PCF update.

6.7.1. Request

A POST request must be sent to the test target host system’s /2/events endpoint with the syntax specified in PACT Tech Specs V2.2 § api-action-events-case-1.

6.7.2. Expected Response

The test target host system must respond with 200 OK and an empty body.

If the test target host system calls the GetFootprint action with the pfId included in the notification, the corresponding PCF must be returned.

6.8. Test Case 019: Notify of PCF Update

This test case will become mandatory test case with the release of version 2.2 of the Technical Specifications.

Tests the target host system’s ability to notify a data recipient of a PCF update.

Note: For this test case, the data owner is the test target host system and the data recipient is the testing party. Accordingly, the latter must be conformant with PACT Tech Specs V2.2 § api-action-events and behave in accordance if the functionality is not implemented.

6.8.1. Request

The test target host system must authenticate with the testing party (performing the customary Authentication Flow and obtain an access token.

The test target host system must send a POST request to the testing party’s /2/events endpoint with a valid access token and the syntax specified in PACT Tech Specs V2.2 § api-action-events-case-1.

6.8.2. Expected Response

If the testing party has implemented the Events functionality, it should respond with 200 OK and an empty body.

Otherwise, it should respond with 400 Bad Request and a JSON body containing the error response NotImplemented, as specified in PACT Tech Specs V2.2 § api-error-responses.

6.9. Test Case 020: Asynchronous PCF Request

This test case will become mandatory test case with the release of version 2.2 of the Technical Specifications.

Tests the target host system’s ability to receive an asynchronous PCF request.

6.9.1. Request

A POST request must be sent to the test target host system’s /2/events endpoint with the syntax specified in PACT Tech Specs V2.2 § api-action-events-case-2-request.

6.9.2. Expected Response

The test target host system must respond with 200 OK.

6.10. Test Case 021: Respond to Asynchronous PCF Request

This test case will become mandatory test case with the release of version 2.2 of the Technical Specifications.

Tests the target host system’s ability to respond to an asynchronous PCF request.

Note: For this test case, the data owner is the test target host system and the data recipient is the testing party. Accordingly, the latter must be conformant with PACT Tech Specs V2.2 § api-action-events and behave in accordance if the functionality is not implemented. This test pressuposes the completion of § 6.9 Test Case 020: Asynchronous PCF Request and should be skipped if it failed.

6.10.1. Request

The test target host system must authenticate with the testing party (performing the customary Authentication Flow) and obtain an access token.

The test target host system must send a POST request to the testing party’s /2/events endpoint with a valid access token and the syntax specified in PACT Tech Specs V2.2 § api-action-events-case-2-response.

6.10.2. Expected Response

If the testing party has implemented the Events functionality, it should respond with 200 OK and an empty body.

Otherwise, it should respond with 400 Bad Request and a JSON body containing the error response NotImplemented, as specified in PACT Tech Specs V2.2 § api-error-responses.

6.11. Test Case 022: Attempt Action Events with Expired Token

Tests the target host system’s ability to reject an Events request with an expired access token with the correct error response.

6.11.1. Request

An Events POST request must be sent to the /2/events endpoint of the test target host system with an expired access token and the syntax specified in PACT Tech Specs V2.2 § api-action-events-request (the EnventBody is irrelevant).

6.11.2. Expected Response

The test target host system must respond with a 401 Unnauthorized and a JSON body that should contain the error response TokenExpired, as specified in PACT Tech Specs V2.2 § api-error-responses.

Note: Since the access token is expired, re-authentication should in principle solve the issue. By returning the HTTP error code 401 (instead of, e.g., 403), the host system signals that re-authentication should be attempted.

6.12. Test Case 023: Attempt Action Events with Invalid Token

Tests the target host system’s ability to reject an Events request with an invalid access token with the correct error response.

6.12.1. Request

An Events POST request must be sent to the /2/events endpoint of the test target host system with an invalid access token and the syntax specified in PACT Tech Specs V2.2 § api-action-events-request (the EnventBody is irrelevant).

Note: Since the access token is invalid, re-authentication cannot solve the issue. By returning the HTTP error code 403 (instead of, e.g., 401), the host system signals that there is no gain in re-authenticating.

6.12.2. Expected Response

The test target host system must respond with a 403 Forbidden and a JSON body containing the error response AccessDenied, as specified in PACT Tech Specs V2.2 § api-error-responses.

Note: Since the access token is invalid, re-authentication cannot solve the issue. By returning the HTTP error code 403 (instead of, e.g., 401), the host system signals that there is no gain in re-authenticating.

6.13. Test Case 024: Attempt Action Events through HTTP (non-HTTPS)

According to PACT Tech Specs V2.2 § api-requirements, a host system must offer its actions under https method only.

Therefore, it is our understanding that host systems must not expose any action-related endpoints through http (non-https).

6.13.1. Request

An http-only equivalent of the test target host system Events endpoint must be generated, replacing "https://" by "http://".

An Events POST request must be sent to the generated http endpoint with the the syntax specified in PACT Tech Specs V2.2 § api-action-events-request (the access token and EnventBody are irrelevant).

6.13.2. Expected Response

No response is expected: the request must not be processed.