In the poem "The Listeners", were the phantoms very close friends with The Traveller that was knocking on their door?

The phantoms are indifferent and completely unresponsive to the Traveller that is knocking at their door - "Never the least stir made the listeners, though every word he spake fell echoing through the shadowiness of the still house" (lines 29-31).  The phantoms can hear the Traveller's cry, but they do not answer, and if we assume that friendship by its very definition involves communication and sharing, then we would have to say that, no, they are not friends at all. 

The theme of the poem is man's inability to connect with others in either the social or the natural world.  With this being the author's message, it is not likely that the phantoms ever were close friends with the Traveller.  Man's ultimate state is isolation, and as such, he has no friends.  Despite his best efforts to establish ties, through the performance of duty as indicated in the lines "Tell them I came...that I kept my word" (lines 27-28) or through other means, man is destined to be alone in a world he does not understand.