Gating ML Compliance

GatingML 2.0 is a flow cytometry standard that is intended to allow for a common representation of flow cytometry gates, and interoperability between different hardware and software vendors. Software that implements the standard must correctly perform several calculations, including gate membership, compensation, and transforms.
Floreada.io has near complete Gating ML 2.0 support.

There are several official compliance tests released with the GatingML standard, which allow for software vendors to test that they are performing all calculations correctly. There is also an implementation of GatingML in R, written by some authors of the standard, which is basically the de facto reference implementation of Gating ML. In the R package they provide additional compliance tests.

Unfortunately for the flow cytometry community, the stated goal of the standard - better software interoperability - has not been realized. Most software vendors, do not support the entire standard, or implement it incorrectly. Even the R implementation, written by the standard's authors does not fully implement it.

At Floreada.io, we believe we have implemented the GatingML 2.0 standard more completely then nearly any other flow cytometry analysis software. Implementing the standard was important to us for two reasons. First, we hope to set a trend that others in the flow cytometry software community follow, and create a better environment for sharing and interoperability between software. And second, by implementing the standard, it allows us to show the accuracy of Floreada.io. In order to pass the multitude of compliance test gates, all gate, compensation, transform, and ratio parameter calculations must be performed correctly and must not be off by even one event in a gate. This is an objective standard for flow cytometry analysis software accuracy which allows us match up the accuracy of Floreada.io against all other analysis software. We think the results speak for themselves.

Official Compliance Tests

There are two sets of official GatingML 2.0 compliance tests which can be obtained here.Each set of tests consists of an FCS file and a gating ML file which should be applied to it. The standard authors also provides what they believe the correct result should be.

We compare the official result for each gate with the result of the gate when applied to the FCS file using various analysis software implementations. Feel free to check the results in Floreada.io yourself. You will need to manually open the FCS file for each compliance set and apply each gate with the appropriate compensation.

Compliance Set 1

The first set of compliance tests all are performed on an FCS2.0 file. To pass all tests, software must be able to:

• Read all the data in the file correctly
• Read all keywords in the file correctly
• Read the gating ML file and generate the test gates
• Apply the appropriate gain to the events in the file
• Apply linear, log, logicle and hyperlog transforms to the data
• Apply rectangle, ellipse, quadrant, and boolean gates and calculate member events
• Apply both in-file and external compensation matrices to the data
• Create and apply ratio parameters

The following table shows the results given by Floreada.io, flowCore - the de facto gatingML reference implementation, and result included in the spec by the spec authors.

1 - When you install the flowCore R package it comes with these GatingML compliance tests. However in the xml file in the Polygon1 gate definition they note that flowCore does not perform the floating point calculations correctly for this gate and so they "cheat" and change the gate definition to get flowCore to come up with the right answer. Basically flowCore messes up the calculation when events are right on the polygon gate and so they change the polygon gate vertices from [(5,5), (500, 5), (500, 500)] to [(5,5), (500, 5), (500, 500), (499.999999, 500), (5, 5.000001)] so a few events are counted correctly. If you test flowCore against the actual compliance tests in the official standard, it gets the wrong answer.

2 - The spreadsheet of results provided in the spec lists the result of this gate as 4710. However flowCore and Floreada.io both get 4770. We think the spec is incorrect. For example, for event #6691, the time value is 26.875, which obviously should be included in a range gate with bounds between 20 and 80, yet the official spec does not include this point in the gate. Given that flowCore, the de facto reference implementation, gets the same result as us, we believe 4770 is the correct answer for this test.

3 - Again quadrant gate results listed in the spreadsheet included in the spec are likely listed incorrectly, as both Floreada and flowCore get a different result.

4 - The AND1 gate test is a boolean gate which should be the result of Range2 and Polygon1 gates. The spec likely has the wrong answer for Range2 (see *2) and flowCore gets the incorrect result for Polygon1 (see *1). Therefore both get this AND gate incorrect as well. If you run the test with flowCore using their "cheat" version of Polygon1, flowCore and Floreada.io get the same result, so we are confident that we are correct.

5 - These gates all use a hyperlog transform that had a typo in the original spec. The results listed in the spec spreadsheet use a hyperlog transform with A = 0. However the actual transform definition in the test file have the Hyperlog transform defined with A = 1. Therefore the spec spreadsheet values are incorrect. The version of the compliance tests including in the flowCore library, make note of this typo in the original compliance tests.

Compliance Set 1 Results

• Floreada.io - 50/50 (100%)

• flowCore - 48/50 (96%)

• Official result 40/50 (90%)

Floreada.io gets the correct result for every gate in the test suite. The flowCore R library, does almost as well. The authors of the software acknowledge in their modified version of the test suite that they have a known bug which causes them to get the wrong result for the Polygon1 gate as well the the AND gate which relies on its result.

Dissapointingly, the spec authors have supplied results to some of the gates which are incorrect. Many of them seem to be the result of typos - such as the noted typo in a hyperlog transform that caused many of the stated results to be off. Regardless of the reason, we hope that the spec authors fix the issues in there results spreadsheet. It is difficult for software to implement a specification correctly without correct information for the specification itself.

Compliance Set 2

The second set of compliance tests all are performed on an FCS3.0 file. To pass all tests, software must be able to:

• Read all the data in the file correctly
• Read all keywords in the file correctly
• Read the gating ML file and generate the test gates
• Apply linear, log, logicle and hyperlog transforms to the data
• Apply rectangle, ellipse, quadrant, and boolean gates and calculate member events
• Apply both in-file and external compensation matrices to the data
• Create and apply ratio parameters

The following table shows the results given by Floreada.io, flowCore, and the results provided by the specification authors