Nearly 80 percent of security products that are sent for certification fail to perform as intended during the initial round of tests, and generally require additional two or more cycles of testing before they are certified, said ICSA Labs.
In a report released Tuesday, ICSA noted that 78 percent of product failures during the first series of tests are due to inadequate performance of core product functionality. A division under Verizon Business, ICSA--which tests and certifies security products--said it based the findings on data collated from over 20 years of product testing.
According to the report, the second most common reason for failures at initial testing is due to the failure to completely and accurately log data, accounting for 58 percent of initial failures. According to ICSA, some vendors and enterprise users consider data logging a nuisance and merely as a "box to check".
The third most common cause of failure is the product's inherent security problems, said ICSA. These problems include vulnerabilities that compromise the confidentiality or integrity of the system, and random behavior that affects product availability.
The study also identified several issues with security products including poor product documentation and problems involving patching--whether a product accepts security updates correctly.
Only 4 percent of products tested at ICSA attain certification in the first testing cycle. However, 82 percent of products resubmitted for testing eventually earn certification, according to the report. ICSA noted that certified products are still required to undergo ongoing tests to maintain their certification.
Play video
Most security products flunk quality tests
So, it links to a 2-year old article stating that "success" was a 100% positive detection rate. I assume that a 0% false-positive detection rate was also considered a requirement for success.
With how frequently new malware is created, this is a completely impossible expectation. Definition-based detection will never be 100%.
Posted by Steve on Wednesday, November 18 2009 06:03 AM