LAS VEGAS—A touch-screen voting machine used in a 2014 election in Virginia was hacked in about 100 minutes by exploiting a Windows XP flaw that was more than a decade old as part of a demonstration on security vulnerabilities in election technology.

The hacker was Carsten Schürmann, an associate professor with IT University of Copenhagen. He was one of the computer hackers invited to the Defcon convention in Las Vegas to test the security and integrity of common pieces of voting technology, many of which were purchased more than a decade ago and are rapidly becoming obsolete.

Within hours of the doors opening Friday at the Voting Machine Hacking Village, hackers penetrated the WinVote 2000 voting machine and gained access to an electronic poll book, the kind used to check in voters at thousands of polling places across the country.Microsoft Corp. made the Windows XP operating system. Microsoft did not immediately respond to a request for comment Saturday.

They also penetrated the hardware and firmware of a kind of touch-screen voting machine used in hundreds of jurisdictions across the country, and could attack a simulated county voter registration network, like the networks in 21 states that were compromised by attackers last year.

Hackers were invited to “do the things that if you did on Election Day, they would arrest you,” said Matt Blaze, a computer science professor at the University of Pennsylvania and election security specialist who helped organize the event.

Fears have mounted among U.S. officials over the security of the U.S. voting system following widespread hacking during the 2016 presidential election.

Last month, Jeanette Manfra, the acting deputy undersecretary for cybersecurity and communications at Department of Homeland Security, told the Senate Intelligence Committee that election-related systems such as voter registration databases in 21 states were targeted last year as part of a Russian government campaign of hacking and disinformation. In that same hearing, former DHS Secretary Jeh Johnson said the hacking, aimed at boosting President Donald Trump at the expense of his Democratic rival Hillary Clinton, was directed by “ Vladimir Putin himself.”

According to a January report from U.S. intelligence agencies, Russia’s tactics included infiltrating and leaking information from party committees and political strategists, and disseminating through social media and other outlets negative stories about Mrs. Clinton and positive ones about Mr. Trump. Russia has denied the report, and Mr. Trump has called the investigation into Russia’s alleged interference a witch hunt.

U.S. officials have said that no evidence has been uncovered that systems involved in vote casting or counting were hacked, though no audits or analyses have yet been performed on the machines used in last year’s presidential election. But researchers say that tampering with touch-screen machines, such as the one Mr. Schürmann hacked, would not be detectable. The WinVote 2000 machine was decertified and not used in last year’s election, Mr. Schürmann said.

“All of these touch-screen machines are unauditable,” said Harri Hursti, another one of the event’s organizers and a specialist in the small field of election technology security. “You should have paper ballots and there should be an audit process.”

After the disputed 2000 presidential election, when a hand recount of Florida’s paper ballots produced weeks of uncertainty about the outcome and criticism of the standards used to count ballots, a new law awarded more than $3 billion to state and local governments to help them replace aging mechanical voting machines that were widespread at the time.

By and large, those governments bought touch-screen electronic voting machines that started to come into service in advance of the 2004 presidential election. But what was once state-of-the-art technology is now antiquated. Many voting systems run on technology that is rarely seen by modern consumers and which has not seen a security update in years.

While there was a great deal of research done into the security of voting machines a decade ago, such as California’s top-to-bottom review of its own voting systems, the pace of research slowed down after 2008, said Mr. Hursti.

The WinVote system hacked at Defcon was likely vulnerable to a wireless attack for years, Mr. Schürmann said. WinVote’s manufacturer, Advanced Voting Solutions Inc. is no longer in business.

The display on the WinVote machine at Defcon indicated that it was used during Fairfax County, Va.’s, Aug. 19, 2014, special election. Attempts to reach Fairfax County after business hours Friday were unsuccessful. The county’s Office of Elections overhauled its voting system in 2014, according to a post on the county’s website.

The Defcon organizers, with a budget of about $20,000, purchased about 30 decommissioned voting machines, including the WinVote 2000, from eBay Inc., several of which had election results stored in the computers.

To hack the WinVote machine, Mr. Schürmann used a pair of widely used hacking tools. First, he connected wirelessly with the machine, a rectangular keyboardless computer that is a little larger than a laptop. Then, taking advantage of a flaw in the voting machine’s Windows XP operating system, he gained administrative control of the system. Microsoft released a software patch for this issue in 2003, but it had not been applied to the voting machine, Mr. Schürmann said.

That allowed him unfettered access to the machine and the ability to manipulate vote results, he said. During the interview, he remotely shut down the voting machine, baffling other conference attendees who had not realized that he had seized control of the system.

Activists for more secure voting are advocating for paper ballots, which are more resilient to remote tampering and technology failures and offer a way to manually recount. While many computerized electronic voting machines now have paper backups, few states audit the results.

“My hope is that there will be momentum for repairing our broken voting systems,” said Barbara Simons, president of Verified Voting, a nonpartisan, nonprofit advocacy group that pushes for paper backups and random audits in voting technology. “There is a solution, and it’s not rocket science.”

** Robert McMillan at Robert.Mcmillan@wsj.com and Byron Tau at byron.tau@wsj.com