One book we'd mentioned on the episode as a counter to Goodman's epistemology was Paul Boghossian's Fear of Knowledge: Against Relativism and Constructivism.
Boghossian's target is any theory of knowledge that says that facts are constructed, reflecting the contingent needs and interest of some society, and that consequently some different society with different needs could construct facts so as to make any given statement that is true according to our current conception false according to the opposite conception.
I think saying only that you should be able to see that this doesn't capture Goodman's view: Yes, facts are constructed according to Goodman, meaning that they're only true relative to a world wherein we build the components for such facts by positing some ontology and standards for judging a statement about that ontology true, and yes, the choice of ontology is not dictated to us by evidence, but neither is it true for Goodman that anything goes, that you could construct a world such that any given statement "true for us" given our conceptual scheme would be false under some other scheme.
Boghossian presents us with the choice between (if you admit any kind of constructivism at all) a Kantian constructivism, where the objects of our experience are constructed in a uniform way according to psychological laws, and a free-form relativism where it's entirely arbitrary how we break down (and/or build up!) the world into describable entities. Goodman's view, in which worldmaking is subject to stringent (though I don't think definable a priori, as this would in effect limit the field of constructible worlds in advance in some too specific way) restrictions, is neither of these.
To attack Goodman, then, he would have to show how Goodman's arguments fail to coherently characterize these restrictions, yet Goodman is only even mentioned four times in Boghossian's slender (not much more than 100 pages) book, and the most sustained argument against Goodman (p. 32-35, criticizing Goodman's essay "Notes on a Well-Made World") picks on a particularly weak example: Goodman cites stellar constellations as a clear case where people have imposed organization on experienced material, and Boghossian rightly points out that this example is in no way typical: constellations are by definition arbitrarily constructed, and that in fact this example shows that there has to be description-independent raw material to be organized, in this case the stars themselves. This clearly does not engage Goodman's fundamental thesis, nor Goodman's pragmatic claim that if there is description-independent stuff out there, then it by definition can't play a role in our epistemic theorizing, as this would require that it be described; this is just the general problem of the Kantian thing-in-itself, which is a pretty big dividing point in the history of (Continental, at least) philosophy.
The fact that Goodman's theory requires that we carve up the world does not imply that there's a world there prior to carving; this is just taking the metaphor too far. A primary step in constructing a world is to posit an ontology, and this is not done by referencing some more basic, primordial ontology and grouping those things into the fundamental entities of your world. Consider the ontologies of physical atomism (where the basic elements are indivisible, physical points) and phenomenal atomism (where a basic element could be "patch of blue here now"). The difference between these isn't captured by saying we construct such an ontology out of something more basic. Instead, we're positing what is basic.
Boghossian's book is very clearly written, and deserves a much more careful account than I'm giving here. Even if it misses its target, it helps tease out exactly what claims the various versions of relativism might be making and what might be wrong with these claims. We will definitely pull this out again in a future epistemology episode focusing on Richard Rorty and/or Hilary Putnam, who are mentioned in the book more often than Goodman.