A protective feature in Windows is locking out the good guys, but letting in a lot of bad guys, according to security software makers.

Microsoft designed PatchGuard to safeguard core parts of Windows, including Vista, against malicious code attacks. But some security companies say that the feature makes it harder for them to protect Windows PCs, as it locks them out of the kernel, the core of the operating system.

"PatchGuard is hurting security vendors more than it is hurting malware writers," Bruce McCorkendale, a chief engineer at Symantec, told CNET News.com in an interview Wednesday. "There are types of security policies and next-generation security products that can only work through some of the mechanisms that PatchGuard prohibits."

Symantec is not alone in its complaints, but it is the largest security company to speak out publicly. Sana Security and Agnitum, two smaller vendors, said they share its concerns, but giants Cisco Systems and McAfee declined to comment for this story.

Microsoft defends the technology, which applies only to 64-bit versions of Windows. Cybercrooks have found ways to exploit the kernel for malicious purposes, making the protection offered by PatchGuard key to securing the operating system, said Stephen Toulouse, a program manager in Microsoft's Security Technology Group.

"It is more important to prevent the installation of malicious software than it is to allow third-party vendors, no matter what the software, to extend the kernel," Toulouse said. "This is not specific to security software. This is a global change to 64-bit Windows to provide a more security computing experience."

Microsoft's push into the security market has put many defense providers on guard. Symantec, especially, looks wary; it has said it will compete with Microsoft as long as there is a level playing field. Now, for the first time, Symantec is saying that Microsoft is limiting the security choices of consumers--which could be interpreted as anticompetitive behavior.

"PatchGuard will make it harder for third parties, particularly host intrusion prevention software, to function in Vista," said Yankee Group analyst Andrew Jaquith. "Third parties have two choices: continue to petition Microsoft to create an approved kernel-hooking interface so products like theirs can work, or use 'black hat' techniques to bypass the restrictions."

Barriers to the kernel
PatchGuard debuted a year ago in Windows XP x64 Edition, but the technology was never broadly adopted. That's set to change when Windows Vista hits store shelves in January, analysts expect. As people buy PCs with 64-bit processors, use of the 64-bit edition of Windows will increase.

In particular, PatchGuard inhibits host intrusion prevention products, security vendors and analysts said. These "HIPS" products are an upcoming class of security software that determines whether a program is malicious by looking at its behavior, rather than using the classic signature-based approach, which checks a program against a database of known threats.

On top of this, PatchGuard blocks features to protect against tampering with security tools, McCorkendale said. Malicious programs increasingly try to disable security software, and the tamper-protection features aim to prevent that.

"There is a whole bunch of companies out there that have pioneered next-generation security, that are limited by PatchGuard," McCorkendale said.

There's another "disturbing side effect," according to a Symantec blog posting. While legitimate security vendors can no longer make extensions to the Vista kernel, attackers have already found ways to disable and work around PatchGuard, it says.

Sana Security and firewall maker Agnitum sounded a similar alarm.

"Bad guys can bypass PatchGuard today," said Vlad Gorelik, chief technology officer at Sana Security, which makes host intrusion prevention software. "Microsoft has this assumption that if you put a shield in, the bad guys will stay out. That is not the way it works. But now they force security vendors to bring a knife to a gun fight."

The barrier to the Windows kernel forces security companies to adopt hacker tactics, Gorelik said. "We will have to come up with alternative mechanisms for doing the same thing," he said. "In some cases, we can actually take a page out of the bad guys' text book and bypass PatchGuard."

Call for exceptions...

With PatchGuard, Microsoft is effectively taking control of security for the Windows core, Gorelik said. Previously, third parties could also provide defenses for that part of the operating system, he said. Now, if PatchGuard breaks, it will be up to Microsoft to fix the flaw and make Windows PCs secure.

"They would have to patch the kernel if someone bypasses PatchGuard," Gorelik said, noting that the kernel is the toughest thing to fix in the operating system.

Security vendors are calling on Microsoft to allow exceptions in the kernel shield for trusted third parties.

"There is definitely a legitimate need to lock down the kernel," McCorkendale said. "I don't suggest they eliminate PatchGuard. What I am asking for is an exception. There are less restrictive means available, and we have proposed many solutions to Microsoft. But it has fallen on deaf ears."

Microsoft opposes the idea of making exceptions, as it would increase the number of entry points that miscreants could take advantage of, Toulouse said.

"When you get into the concept of exceptions, you get on a slippery slope," he said. "What made a lot of sense to us is simply to restrict the kernel without exception, creating a level playing field that all of the vendors, including Microsoft, can then operate by." Toulouse's argument is that Microsoft's security software is also unable to touch the kernel.

Dropped ball
With the advent of threats such as rootkits, which that nestle deep inside the operating system, Microsoft should protect the Windows core, analysts said. However, the company has dropped the ball on letting other software makers in on what the new kernel protections mean for them, said John Pescatore, an analyst at Gartner.

"This is a complex issue, but Microsoft has definitely been deficient in including the impacted software makers early on," Pescatore said. "That definitely does work to their advantage from a competitive viewpoint. However, the rootkit issue has to be fixed, and kernel protection has to be stronger for all operating systems."

Indeed, Symantec is playing the anticompetitive card for the first time. The Cupertino, Calif.-based company had said it would beat Microsoft by using its security wits as long as the competition is fair. Now the fairness seems to be gone, McCorkendale said.

"It seems a bit disingenuous of Microsoft. They are getting into the security market and are disallowing this whole class of security products that they don't have," McCorkendale said. "It does not feel like a level playing field at that point."

McCorkendale stopped short of saying that Symantec would sue Microsoft or complain to antitrust authorities. However, Yankee Group analyst Jaquith believes that step is getting closer, especially if Microsoft were to give its own security products a way to bypass PatchGuard.

"Microsoft's anti-kernel hacking feature could conceivably create a formidable barrier to entry to their competitors in the security market," Jaquith said. He expects Microsoft to deliver host intrusion prevention capabilities in its Forefront products next year.

"I think you'll see the larger security companies run to the Department of Justice and the European Union faster than you can say 'Penfield Jackson'," Jaquith said, referring to Thomas Penfield Jackson, the judge who oversaw the landmark U.S. antitrust case against Microsoft.