WASHINGTON -- Members of the House Intelligence Committee are declining to stick up for their colleagues' anti-Muslim allegations, saying that their conduct is unbecoming of someone in their position with access to classified information. None, however, is going so far as to call for them to give up their seat on the committee.

"We have a small committee, we work hard together, it's a very bipartisan committee and we deal with national threats," said Rep. C.A. Dutch Ruppersberger (D-Md.), the ranking member on the House Intelligence Committee. "This does not help our committee at all."

Last week, committee chair Rep. Mike Rogers (R-Mich.), a former FBI agent, similarly dismissed the members' allegations that the Muslim Brotherhood may be infiltrating the U.S. government.

"That kind of assertion certainly doesn't comport with the Intelligence Committee , and I can say that on the record," he told USA Today.

Reps. Michele Bachmann (R-Minn.), Trent Franks (R-Ariz.), Louie Gohmert (R-Texas), Thomas Rooney (R-Fla.) and Lynn Westmoreland (R-Ga.) recently wrote letters to the inspectors general at five federal agencies asking them to investigate the influence of the Muslim Brotherhood. Their most high-profile target was Huma Abedin, a top aide to Secretary of State Hillary Clinton. They alleged that Abedin may have ties to the Brotherhood through her family.

Bachmann, Rooney and Westmoreland are all members of the Intelligence Committee and therefore have access to classified information.

"Given our access to sensitive information, I also believe members of the Intelligence Committee have a special responsibility to exercise caution in making statements about national security concerns," said Rep. Jim Langevin (D-R.I.), a member of the committee. "The only reasonable action for the authors of these letters to take would be to withdraw their requests."

"The unfounded allegations made by some members of the Intelligence Committee against Huma Abedin are deeply disturbing -- and damaging to the committee's work and its reputation. The authors discredit themselves and are deserving of no further comment," added Rep. Adam Schiff (D-Calif.).

The progressive group People for the American Way has called on House Speaker John Boehner (R-Ohio) to remove Bachmann from the Intelligence Committee.

Members of the House Intelligence Committee are entrusted with classified information that affects the safety and security of all Americans," said PFAW President Michael Keegan. "That information should not be in the hands of anyone with such a disregard for honesty, misunderstanding of national security, and lack of respect for her fellow public servants."

Boehner, however, has indicated no desire to reconsider her position.

"I don't know that that's related at all, " he said when reporters asked him last week whether he would remove her.

Ruppersberger declined to give his opinion on the matter, saying the decision rested with Boehner.

"I don't really want to get into that because it's not my call," he said. "I just know I'm concerned about the comments. I think that those of us in leadership have said that the comments were not factually correct, and we think that it was not the right thing to say. It was wrong. You don't want to discriminate against any American. If the allegations in there [their letters] were true, believe me, we would be dealing with it -- our law enforcement agencies would be dealing with it."

Bachmann has maintained that her concerns have been "distorted ." On Tuesday, Gohmert called his critics "numb nuts ."

Lawmakers aren't the only ones thinking their five colleagues are fabricating conspiracy theories -- the Muslim Brotherhood is also confused.

Ahmed Al Nahhas, an activist and leader of the Brotherhood in Egypt told the GlobalPost that the allegations were "ridiculous ."

Ibrahim Ali Iraqi, another leader in Egypt with the group, joked that they "can't even penetrate the Egyptian government ."