Mark Allen, the former MI6 officer at the centre of the row over British links with Libya, masterminded the mission to persuade Muammar Gaddafi to surrender his weapons of mass destruction and escape the fate of Saddam Hussein in Iraq.

But the plaudits he won for his work in 2003 now risk being overshadowed by documents that have surfaced in Tripoli showing British agencies' close ties with Libyan intelligence and raise troubling questions about UK complicity in illegal "renditions" of terrorist suspects.

Allen, a slight, academic-looking figure, was MI6 director for counter-proliferation and counter-terrorism when the secret initiative he undertook with the CIA brought Gaddafi in from the cold, paving the way for a western-Libyan honeymoon – until the Benghazi uprising in February.

Allen, who has spent his professional life in the shadows, has declined all requests to comment – on or off the record – about his chummy letters to Moussa Koussa, who was Libya's foreign minister and security supremo until his dramatic defection in March.

But those who know Allen describe a smooth and highly intelligent man who was uniquely placed to do business with the Gaddafi regime. "Tough, charming, hardnosed and extremely clever," is the verdict of one acquaintance. "Loyalty means a lot to Mark," said another ex-MI6 man. And, for most spooks, loyalty means absolute silence.

No one calls him Allen of Arabia, although Arabs have played a huge part in his life. In 1980, he published a highly regarded book on Arabian falconry with a preface by the great explorer Wilfred Thesiger. Last year he edited a book on Arabic calligraphy – another passion he lists as a hobby in his otherwise terse Who's Who entry.

Now 61, Allen studied Arabic at Oxford on a Foreign Office scholarship and then spent a year living with bedouin in Jordan, where he owned a thoroughbred camel. After that he went on to Mecas, the "school for spies" in Lebanon, where his former teacher, Leslie McLoughlin, still remembers him as a "very bright student". In his short 2006 book, Arabs, he describes the difficulty of learning the language – though his fluency is "amazing", according to an envious acquaintance.

Allen's first MI6 posting was to Abu Dhabi in 1975. He remains well-connected in the Gulf and turned heads at a conference in Bahrain by embracing a succession of Arab intelligence chiefs – saying he had "trained the people who trained them".

In the late 1970s he was stationed in Cairo, watching the Iranian revolution, the Egyptian-Israeli peace treaty and the assassination of Anwar Sadat by Islamist militants. By 1990 he was head of MI6's Middle East station in Jordan at the time of the first Gulf war. He was close to King Hussein. Iraq was his main preoccupation.

From 1994-2004 he filled increasingly senior jobs at MI6 HQ in London but left after failing to get the top job, which went to John Scarlett. His knighthood in 2005 was seen as grateful recognition of his Libyan WMD coup – when Gaddafi dismantled his arsenal as the price for ending his pariah status.

Friends say Allen's Catholicism is extremely important to him, as evidenced by two books he has written on prayer and holy communion.

"Mark is so charming and clever and he's not short on technique either," says one admirer. "He would say 'Let's have a drink … and that's when gossip turns into information." Allen is said to be fascinated with power.

"Everything about him says man of culture, who is slightly eccentric, with a very British way of expressing himself," notes another old Middle East hand. "He looks too small, skinny and slight to be a spy. He does not look as if he is involved in anything cloak and dagger."

Allen was adamantly opposed to the war in Iraq. "Mark stuck his head out of the window and his fingers in his ears until the planning was over," a colleague recalled. "He thought it was an absolute disaster."

Named only as witness SIS4 – but easily identified by his witty style and Latin quips – he told the Chilcot inquiry of "endless conversations" inside the agency as political pressure built in Whitehall to produce a "silver bullet" on Iraqi WMD.

"The idea that Iraqi Shias could be fitted out with Republican, Democrat, Lib Dem identities, organisations and run the difficult place which is Iraq wouldn't have occurred to me in 2001," he said.

Allen gave a memorable answer when asked if he thought MI6 had got too close to policy-makers. "I don't think … that we got too close to the sun," he said. "The Icarus metaphor is used time and again. It has limited applicability because Tony Blair was not the sun and [MI6 chief Richard] Dearlove was not a child with wax wings. They were consenting adults, wrestling with unprecedented policy riddles."

Scepticism about prospects for a stable post-Saddam Iraq continued in his post-MI6 job as an adviser to BP. And his Libyan contacts were useful in his role with the Monitor Group, a consulting firm in the US that was heavily involved in Tripoli. Allen's name also surfaced in the controversy over the 2009 release of the Lockerbie bomber, Abdelbaset al-Megrahi. In that period he escorted Gaddafi's son Saif al-Islam to meetings in Oxford.

Allen's 30-year career in the secret world looks likely to be most remembered for his involvement with Libya – although not for the reasons he might want.

"Libya was his triumph, his crowning glory," said a British diplomat. "He actually got them to give up their chemical weapons. OK, so you keep up a slightly slimy relationship with people like Moussa Koussa and you write letters that might in retrospect have been a bit fawning and silly – but in the end they did do exactly what they were supposed to do."