I almost don’t want to say anything more than what is in the title for fear it will influence the answers. I guess I would just add that, whatever your definition is, do you like it?
This is a hard question!
My personal opinion is that genres are never strict segregations. Ultimately, when we publish a book, we have to select three categories that we think the book best fits in, but very few books super cleanly fit just one box. We always have to push it over the line in one direction or the other.
When I think of literary fiction, I think of stories that emphasize character over action. There has to be action for it to be a story, but really we evaluate the protagonist winning or losing based on how they develop and change as a person. Literary fiction often has a heightened emphasis on internal conflict.
But because genres are fluid, I think that it’s entirely possible to have literary sci-fi, literary fantasy, literary thrillers, etc. even though “genre fiction” (so to speak) is often viewed as the opposite of literary fiction.
Fellow Travellers is a super cool book that I had the privilege to develop and publish a few years ago: https://amzn.to/48AWXW2 We fell into that predicament. It really is a literary time travel book. It has all of the typical sci-fi time travel elements, but character development really carries the book.
I don’t think this is an objective, right or wrong question, so these are just my two cents. I’d love to hear some other perspectives, too.