PACT Conformance Testing

Living Document,

This version:
https://wbcsd.github.io/pact-conformance-testing/
Feedback:
public-dev@pathfinder.sine.dev with subject line “[pact-conf-test] … message topic …
Issue Tracking:
GitHub
Editor:

Abstract

This document provides the general guidelines for conformance testing within the PACT ecosystem.

1. Context

The Partnership for Carbon Transparency (PACT) initiative is continuously evolving and bringing together organizations and industry initiatives to exchange accurate and primary data-based Product (Carbon) Footprints irrespective of geography and industry. The technical foundation for this exchange is currently the Pathfinder Framework and Technical Specifications for achieving interoperability.

If a company wants to exchange Product Footprint information in a truly interoperable way, it then either needs to implement these foundations or look for an existing solution. In both cases, companies need to be able to assess the conformance of their software and solutions.

This document defines the overall process of testing the conformance of software implementations, and the requirements for having a solution listed publicly online on the PACT Online Catalog.

2. PACT Conformance Testing Overview

Conformance testing is a process of verifying if a product or service meets specific standards or specifications. In the context of PACT this is the process of verifying that a solution or system implements the Pathfinder Framework and technical specifications.

As the technical and solution landscape in the PACT context is rapidly evolving, in a first step a very “lean” and minimal process is defined. This process will evolve further in scope and complexity, and include software-based automation in the future.

What is a PACT Conformant Solution?
A PACT conformant solution “A” is defined as a solution which:

Was tested for conformity with a specific version of the technical specifications with 2 or more other solutions

Where the other solutions independently confirm the conformance of solution “A”

2.1. What is the focus and scope of PACT conformance testing?

The PACT Conformance testing process focuses on testing the technical interoperability of a specific implementation (called “host system under test”):

A. Technical interoperability testing is scoped by data exchange protocol version:
an implementation is tested for conformance against a specific version of the data exchange protocol (such as version 1.0.1 or a later version such as version 2) by a 3rd party.

B. Technical interoperability testing focuses on the HTTP REST API:
for this, the 3rd party makes conforming calls to the HTTP REST API of the host system under test. The 3rd party then compares the witnessed behavior with respective specification.

C. Optionally, technical interoperability testing covers interoperability testing of a data model extension:
for this, the parties test technical interoperability against a specific data model extension version. If the data model extension provides instructions for doing so, the testing process shall cover them.

2.2. When does a host system under test satisfy a PACT conformance test?

A host system under test has satisfied a PACT conformance test if the 3rd party declares it has performed a PACT conformance test with the host system under test, and that the behaviour of the host system under test was conforming to the specification (see e.g. § 4.2 PACT Conformance Test criteria (Checklist) of this document for a checklist).

2.3. What is not yet in scope of the PACT conformance testing?

For the first iteration of the Conformance Testing - testing of conformance with the Pathfinder Framework, GHG Protocol (likewise ISO 14044 etc.), sector-specific or product-specific rules is not in scope.

The first iteration of Conformance Testing will validate solutions conform to the PACT technical specifications only. Solutions will not be assessed for conformance to Product Carbon Footprint methodologies, such as Pathfinder Framework, GHG Protocol (likewise ISO 14044 etc.), sector-specific or product-specific methodologies.

If a host system under test implements the HTTP REST API but contains a (critical) bug in its GHG protocol implementation, the solution would still meet the PACT conformance testing requirements.

Note: Future definitions of the conformance testing process will extend their scope.

3. Connectathons

A Connectathon is an event organized by WBCSD PACT. It is geared towards simplifying interoperability and conformance testing between implementers and their respective solutions.

A Connectathon is open for invited parties and logically consists of

  • Key Date when the Connectathon takes place and collaboration starts
  • List of companies participating
  • Dedicated MS Teams channel to collaborate and share feedback, test progress, and outcome

    3.1. Connectathon Process

    Organizations can Onboard their Solution to PACT Online Catalog before or after the Connectathon.

    Please refer the process below -

    Connectathon process diagram
    Connectathon process diagram

    Note: Connectathons will be conducted on a Monthly basis

    4. Conducting a PACT Conformance Testing

    Conformance Testing commences once Organizations identify their Testing partner(s) before or during Connectathons.

    4.1. Process

    The key stages of a PACT Conformance Test performed by the parties are depicted below:

    Conformance process diagram
    Conformance process diagram
  • Organizations will Initiate the Conformance Testing process by aligning on Timeline, Process, and Communication mechanism (dedicated Private MS Teams Channel).
  • Conformance Test begins based on the Test Data / Environment setup initially. It is assumed, the parties perform the testing in accordance with the [PACT Criteria Checklist] for the testing. The parties are free to use Teams or different channels during the Connectathon or after. Additionally, the parties are free to use the PACT-sponsored SharePoint to share data among each other.
  • Organizations will continuously / iteratively run test scenarios and evaluate the PCF Data exchange as per the PACT Conformance testing criteria Checklist. Issues / queries can be posted and resolved via the MS Teams Channel. After successful conclusion of a conformance test, Organizations will Sign Off the Solutions (Pass/Fail)
  • Organizations are encouraged to retest their solutions during the next scheduled Connectathon, especially in case when issues have been resolved or support for new versions (related to the data exchange protocol or 1 or more data model extensions) was added to a previously-tested implementation
  • Organizations are encouraged to onboard their solution to the PACT Catalog and to upload conformance results there as well (see here for details)
  • The Conformance Testing process is periodically reviewed, updated, and scope expanded (see § 2.3 What is not yet in scope of the PACT conformance testing?) to reflect changes in the organization’s standards and requirements, as well as feedback received from members of the network.

    4.2. PACT Conformance Test criteria (Checklist)

    Recommendation from WBCSD to be used as a reference for the Bilateral Conformance Testing criteria for Technical Interoperability. A Solution is deemed PACT Conformant if it meets all the criteria’s as defined in this checklist.

    5. Onboarding Solutions to the PACT Online Catalog

    Companies can onboard their Solutions to PACT Online Catalog at any time – regardless if their Solutions were tested for Conformance or not tested yet. Depending on whether the Solution was Conformance tested or not, the Status of the Solution in the PACT Online Catalog will reflect accordingly once they are onboarded to it. Please Refer below -

    Onboarding process diagram
    Onboarding process diagram

    5.1. Process to Submit a Solution in Online Catalog

    1. Navigate to the “Contribute” section
    2. Click the first “Go to GitHub” button under the “Developing Solutions?” option
      • You will be taken directly to the WBCSD PACT Online Catalog repo.
      • Follow the process as described in the CONTRIB_SOLUTION.md to Submit a Pull Request (PR) for your Solution
    3. Once the Pull Request is submitted (above step), the Pull Request will be reviewed for completeness. If all details are filled out correctly, the PR will be merged and the Conforming Solution will appear under the “Solutions" page on the Online Catalog
    4. After clicking a particular “Conformant Solution”, you will be taken to its Details page (example)
    5. You can review the details of all your Conformance Tests under the “Conformance” tab (example)

    6. FAQs

    Do participants need to identify a partner BEFORE the Connectathon kickoff?

    PACT Team will conduct a Kickoff before every Connectathon which will cover – Overview of the Conformance Testing Process, Introduction of the participants, Logistics (MS Teams channel, Communication / collaboration mechanisms between testing partners), Demo of the Online Catalog and How to add Conformance Test Results and Solution to the Catalog, and any Q&A. Participants can decide during this kickoff or post Kickoff who they want to collaborate with for the Conformance Testing

    What are the pre-requisites for participating in the Connectathons?

    The main pre-requisites for participating in the Connectathons is that you have a Solution (fully ready or partially ready) that follows Technical Specification V1 or V2

    Is there a limit on the Testing Partners an Organizations can conduct Conformance Testing with during the Connectathons?

    Organizations are welcomed and encouraged to pair-up with multiple Partners who are available for Testing. Note: Minimum 2 independent Organizations need to conform the Solution for the Solution to be deemed PACT Conformant

    How many times can we offer up our solution to be tested in Connectathons?/How many Connectathons can we participate in?

    Organizations are welcomed and encouraged to participate in multiple Connectathons and conduct multiple Conformance tests within a Connectathon.

    Is there a component of "learning" as part of Connectathons?

    Once the conformance Testing is concluded, Organizations are recommended to share the Test Results and the observations. (In the format they choose to). These observations can be leveraged as learning opportunities. PACT Team will also plan on organizing Retrospective sessions after a Connectathon is concluded, so that Organizations who participate in the Connectathons and conformance testing can share their experience and learning.

    Our Solution passed the Conformance Tests during a Connectathon. Can we still participate in any upcoming Connectathon?

    Yes you can participate in any / every Connectathon based on your availability and readiness.

    Can Organizations participating in the Connectathon collaborate over their preferred Communication Platform or communication via WBCSD MS Teams is a mandatory requirement?

    Organizations are welcome to use their preferred Communication platform based on the alignment with their Testing Partner(s).

    When should we upload the Conformance Results to the online Catalog?

    Organizations are encouraged to upload the Conformance Test results to Online Catalog if the Solution passes the Conformance test. If the Solution fails the Conformance Test, it is up to the Organizations to upload that Result to the Online Catalog and it is not mandatory.

    7. Resources

  • Pathfinder Technical Specifications Version 1.0.1
  • Pathfinder Technical Specifications Version 2.0.1
  • PACT Online Catalog
  • Instructions on how to publish Conformance Test Results to the PACT Online Catalog