What is the meaning of self consciousness applied to postmodern literature (short story, novel, etc.)?

In many respects, self-consciousness is one of the central tenets of postmodernism. Simply put, self-consciousness is the process by which a postmodernist work, whether it's a short story or novel, shows the reader that it is aware that it is a work of fiction. This differs from conventional works of fiction, which are generally meant to be read as if they were actually real, though the reader knows that it is make-believe. 

Understanding this concept is easier with an example. Take, for instance, episode fifteen of Ulysses by James Joyce. Though the novel is generally considered a modernist work, it actually exhibits several postmodernist characteristics, especially in the fifteenth chapter. In this sequence, which is written in the style of a play script, Leopold Bloom wanders through the red light district of Dublin and experiences a great deal of hallucinations involving characters encountered previously in the book. These experiences are excessively outlandish, as they involve impossible events, such as Bloom's establishment of a fictional city called Bloomusalem, talking soap, and an encounter with the End of the World, which is personified as a two-headed octopus with a Scottish accent. The absurdity of these visions, along with the overly artificial style of the form of the text, brings attention to the fact that you are, in fact, reading a work of fiction. Moreover, many scholars take these hallucinations not to be the product of Bloom, but rather of the book itself. The novel's self-consciousness is most evident in this chapter because all of the themes, characters, and ideas it has wrestled with come to the surface, bringing attention to the book's artificial consciousness. This is a prime example of postmodernism's concept of self-consciousness.