American heavyweights Paul Auster and George Saunders are to go head to head for this year’s Man Booker prize, as major names make way for two new faces on the 2017 shortlist.

The judges, chaired by Baroness Lola Young, announced their shortlist of six titles on Wednesday morning. Alongside Auster and Saunders, the 29-year-old British debut novelist Fiona Mozley has secured a place in the final line-up, as did American first-timer Emily Fridlund.

The two debuts are up against two previously shortlisted authors: Scottish author Ali Smith, who is in contention for a fourth time with her post-Brexit novel Autumn, and UK/Pakistani writer Mohsin Hamid’s novel Exit West, in which refugees can use strange black doors to escape to other parts of the world. Hamid was previously shortlisted for his 2007 novel The Reluctant Fundamentalist.

But a host of award-winning writers failed to make the cut, with former Booker winner Arundhati Roy missing out on a place, as well as Sebastian Barry, Kamila Shamsie and Mike McCormack. British authors Zadie Smith and Jon McGregor also dropped out.

American author Colson Whitehead’s The Underground Railroad was another high-profile casualty. It was the bookies’ favourite, and had already won a string prestigious literary gongs, including the 2017 Pulitzer prize for fiction, the 2016 National Book Award for fiction and the Arthur C Clarke award for science fiction.

Waterstones fiction buyer Chris White was among those shocked by its absence. “We’re all used by now to the Booker judges delivering surprises but the omission of The Underground Railroad from the final six certainly ranks among the biggest shocks I’ve witnessed. I think that, when we look back at 2017, we may see this as the one which got away,” he said.

Instead, judges selected short-story writer Saunders’s first novel Lincoln in the Bardo, which follows US president Abraham Lincoln as he visits the grave of his son Willie, and Auster’s 4321, a novel that judges called “magisterial”, about a boy called Archibald Isaac Ferguson, whose life takes four simultaneous fictional paths.

Mozley was picked for her first novel, the story of a man and his children who retreat to live in a copse in Yorkshire’s West Riding, Elmet. Her fellow debut, the American novelist Emily Fridlund’s History of Wolves, focuses on an isolated 14-year-old as she grows up in a failing cult in the US’s midwest.

Young described the shortlisted titles as “unique and intrepid books that collectively push against the borders of convention”.

“Playful, sincere, unsettling, fierce: here is a group of novels grown from tradition but also radical and contemporary. The emotional, cultural, political and intellectual range of these books is remarkable, and the ways in which they challenge our thinking is a testament to the power of literature,” she said.

But with half of the authors from the US, judges were questioned at a press conference about the “Americanisation” of the UK’s top literary award. The £50,000 prize was opened up to US writers three years ago, and was won last year by American Paul Beatty for The Sellout.

Ladbrokes immediately named Saunders the favourite to win the prize at 2/1. Hamid and Mozley came in second, at 4/1, while Auster was given odds of 5/1, and Fridlund and Smith both at 6/1.

Her fellow judge, the literary critic Lila Azam Zanganeh, added that less than 30% of the books submitted for the prize were by US writers, a drop on the previous year. “I feel we are transcultural, increasingly,” she said.

The novelist Sarah Hall, who was also on the judging panel, said that an element that had emerged from each of the shortlisted titles was “the idea of liminal spaces, whether that’s moving doors, or taking down the walls of consciousness, or life”.

“Going from the longlist to the shortlist was tough,” said judge and travel writer Colin Thubron. “There were several novels one judge felt strongly about which they were gracious enough to drop in [the face of] opposition from others. We had give and take. But in general there is no book on the shortlist which I feel shouldn’t be there.”

The panel, completed with artist Tom Phillips, took six hours to narrow the field, in what was a “pretty robust discussion” according to Young.

“There’s no such thing as a perfect novel,” said Hall. “So as the books sustain the tests of technicality, of interiority, of strength of character, it becomes harder [to choose], because what is a perfect novel?”

The winner will be announced on 17 October.

The 2017 Man Booker shortlist

4321 by Paul Auster (Faber & Faber)

History of Wolves by Emily Fridlund (Weidenfeld & Nicolson)

Exit West by Mohsin Hamid (Hamish Hamilton)

Elmet by Fiona Mozley (JM Originals, John Murray)

Lincoln in the Bardo by George Saunders (Bloomsbury)

Autumn by Ali Smith (Hamish Hamilton