The Federal Police was supposed to be this country's answer to the FBI—a tough investigative and crime-fighting force.

But when Federal Police officers allegedly tried to kill two U.S. government employees outside this hillside village last month, doubts surfaced on both sides of the border about whether the agency can be trusted to lead Mexico's charge against organized crime.

Mexican and U.S. investigators continue to sort through clues to reconstruct the Aug. 24 attack outside the capital that wounded two U.S. officials, which the U.S. called an ambush by Federal Police. Mexican officials haven't classified the attack, but say they are holding a dozen federal police officers as investigators prepare charges against them for abuse of authority and attempted homicide.

The Federal Police said last week the officers were in the area investigating a kidnapping that took place hours earlier and mistook the car for kidnappers when they shot at it. The agency didn't say why the officers allegedly ambushed a SUV with diplomatic plates and issued no apparent warning before shooting.

"The first thing we heard was gunfire coming our way," said a witness at a gas station where the U.S. officials briefly took refuge during the 20-minute ordeal.

A U.S. government official said while the incident was disturbing, he was confident that Mexican authorities were investigating it and the shooting would not derail U.S.-Mexico cooperation in fighting crime groups. The office of President Felipe Calderón and the Federal Police declined to comment.

It was the latest high-profile incident involving Federal Police under Mr. Calderón, whose terms ends in December. In June, three Federal Police officers were killed by their own colleagues in a shootout at a food court in Mexico City's airport.

Federal Police said the shootout happened when one group of their officers were sent to arrest colleagues believed to be running a drug-trafficking ring inside the airport. Two of the alleged ringleaders escaped the heavily guarded airport and remain at large. The agency reassigned all 350 officers working there.

In 2010, the agency fired 3,200 officers—10% of the police force—for failing lie detector tests and other checks meant to guard against corruption.

That didn't stop the controversies. That same year, a group of officers in Ciudad Juárez, a town on the U.S. border, organized a mutiny against their commander, telling newspapers he was running extortion rings.

The incidents are an embarrassing setback for Mr. Calderón, who has hailed the creation of the Federal Police as the future of Mexican policing and one of his biggest accomplishments since taking power in 2006. His government created the force three years ago out of a smaller, existing agency and it was meant to be a trustworthy counterbalance to Mexico's many local and state police forces, which have been mired in corruption.

"Mexico has seen the constant creation and recreation of police forces—it has resulted in a lot of new acronyms but not better policing," said David Shirk, who has studied Mexican policing at the University of San Diego's Trans-Border Institute. "They want to create the new untouchable police entity. So far they haven't."

It is unclear where the force will head next under Mexico's next administration. President-elect Enrique Peña Nieto hasn't announced any big changes to the Federal Police, but has proposed plans to beef up security by forming a new national "gendarmerie," which would perform many police duties but would draw its ranks from ex-soldiers.

Some Mexican officials see the Federal Police as a source of crime rather than its solution.

"Whenever the Federal Police come to Nuevo León [state], we'd get more extortions and kidnappings," said a former top state official in the northern border state that has been battered by drug violence. The official said the police weren't trusted by other security agencies, particularly the Mexican navy, which is widely considered the country's cleanest force. "The Federal Police is a disaster," he said.

Last week, the Federal Police suffered an embarrassment after it spent more than $4 million to build a museum to celebrate its history—something local papers ridiculed given its short tenure and apparent need to invest in its own human capital. The agency hasn't responded to the criticism.

Other attempts to tout the agency have also backfired. Last year, Mexican broadcaster Grupo Televisa S.A.B. filmed a series about the police meant to boost public opinion of them—a move that caused an uproar when the agency acknowledged that the Public Security Ministry, which controls the force had used $10 million of public funds to pay for the show. At the time, the government said it had acted transparently in an effort to raise public awareness.

Despite the growing pains, many experts say the Federal Police is the best alternative to Mexico's fractured landscape of police officers, whose low pay makes them easy targets of crime groups.

The force is led by Mexico's Public Security Minister Genaro García Luna, who requires new recruits to have at least a high-school diploma and investigators to have college degrees. He gave station commanders bulletproof vests and modern cars and raised salaries to an average of $525 a month from about $300, said Ernesto López Portillo, who studies the federal police at Mexico City think tank the Institute for Security and Democracy.

Since 2009, the force has grown by seven times to 35,000 today—an expansion that may have created an opportunity for drug traffickers to put their own agents on the force, analysts say.

The agency relied heavily on the use of lie detector tests to vet new recruits. But many police commanders felt the Mexican officials in charge of the polygraph tests weren't well enough trained, said Daniel Sabet, a Georgetown University researcher who was invited by the federal police to evaluate its disciplinary system last year.

In one case, Mr. Sabet said he interviewed a municipal police commander who sent two groups of police for testing, one group of clean officers and a second group of ones the commander believed to be corrupt—only to find that many suspected corrupt officers were passing the tests.

"The Federal Police put all their eggs in the basket of the polygraph tests, and this was a problem," Mr. Sabet said.

The Federal Police have in the past defended their system of lie-detector tests and vetting as helping to keep their force clean.

Mr. García Luna has long said its polygraph tests are accurate and point to other purges since 2010 as evidence of their efficacy.

While vetting new recruits is important, the police also lack a robust internal-affairs department, said Eric L. Olson, who has studied the Mexican Federal Police at the Washington-based think tank Wilson Center.

Mr. Olsen says that as of last year, roughly 95% of the internal investigations under way at the agency involved administrative issues like officers not wearing uniforms or showing up to work. The agency had recently set up a program to catch corrupt cops through sting operations, but had only employed it twice—in both cases to catch police who were asking for minor bribes on roads outside of the industrial hub of Monterrey, Mr. Olson said.

The training of the new police officers is also suspect, Mr. López Portillo said. In the case of the Mexico City airport shooting, police made "nearly every mistake possible," said Mr. López Portillo, including not establishing a perimeter around the suspects to prevent their escape to trying to apprehend them in a food court filled with travelers.

The Federal Police has said it is reviewing video footage of the incident to reconstruct what happened that day. In the past, it has defended the training of its officers as among the country's best and notes that it works with advisers from the U.S. and Colombia in training programs.

*A Troubled Beat

Mexico's Federal Police has suffered irregularities

August 2010 - Officers stage a mutiny in Ciudad Juárez, alleging that elements of the force are running extortion rackets in the northern border city.

July 2011 - The Federal Police are discovered to have paid $10 million to broadcaster Televisa to produce a favorable show about the police force.

June 2012 - Officers open fire in a crowded airport terminal killing three fellow officers during what they said was an operation aimed against corrupt police in their ranks.

AugustFederal Police shoot two U.S. officials in the hills outside of Mexico City. Mexico takes 12 officers into custody.

*Jose de Córdoba contributed to this article.

**Nicholas Casey at nicholas.casey@wsj.com