Sometimes I do so because the gender is irrelevant

This seems to be part of the pathway of change that has led to the widespread adoption of specific singular “they” among younger speakers, and there’s some empirical evidence supporting this.

What I don’t get is, how can knowing the gender suddenly make it difficult to use a neutral term, if it worked before?

This is just one of those arbitrary rules that often exist in language, like how in many languages neuter/inanimate nouns can’t act as the subject of a sentence due to what’s called an “animacy restriction”.

For this specific phenomenon (the “older” ungrammaticality of definite singular they vs. the “younger” grammaticality of it), this recent paper argues that this is due to a difference in obligatoriness of morphosyntactic gender features. The paper is a bit technical if you don’t have a linguistic background, but the basic argument is that in older varieties of English, gender features must be obligatorily expressed in the morphosyntactic derivation if they are known, while in younger varieties, this expression seems to be optional, and therefore free variation between he/they and she/they is allowed by the grammar.

So, “It’s John’s. They need to come get it” is ungrammatical for older speakers for not obligatorily expressing the gender feature once it’s known, while it’s perfectly fine for younger speakers for whom expressing that feature seems to be optional in the grammar.

Maybe this analogy will help: Let’s say you meet someone, and you ask them “Do you have a cat?”. Note that you’ve used the singular here, though it’s acting number-neutral in this context. If they respond “I have two”, then it will immediately become ungrammatical for you to continue to use the number-neutral singular and ask “Does your cat like fish?”

Once you have access to the information that there’s more than one cat, then the arbitrary rules of English grammar require that knowledge to immediately be reflected in the morphosyntactic structure of your sentences from then on. And this makes no independent, logical sense, because there are tons of languages out there that don’t have plurality distinctions. But, English does, and so to speak grammatical English (for now), you have to use plural morphology to refer to more than one entity.

It’s the same for “older” speakers of English - just like it’s ungrammatical for you to continue to use the number-neutral singular once you know that there’s a “plural number feature” in the linguistic context, for older speakers of English it’s ungrammatical for them to continue to use the gender-neutral “they” once they know that there’s a “masculine gender feature” in the linguistic context.

Also, it’s important to note that this term “ungrammatical” is descriptive, not prescriptive - it’s not saying that it’s not “proper” or “correct” according to some arbitrary standard that someone decided on in the 1800s, but rather that’s literally not how those speakers’ mental grammars work. While it may seem illogical (and even regressive from a modern political perspective), every natural human language is composed of arbitrary rules that often seem illogical. Like how the past tense of “go” is the completely unrelated past tense of the older English verb “to wend”, “went”. Or how the past tense of the verb “can” isn’t “could” anymore – that’s reserved for modal usage now in most English dialects – it’s the completely awkward phrase “was able to”.

That doesn’t mean that we can’t, or shouldn’t, try to accommodate non-binary people of course, as is unfortunately often argued, but it does mean that, contrary to what I commonly see people say on the internet, doing so for these speakers does require a constant, concerted effort to consciously override their mental grammars.