Yes, they're on topic and should remain so. We can answer them effectively and can get pretty satisfying answers to these questions that are good examples of our quality guidelines. We agree they're relevant to RPG expertise and they get answered based on a mix of subjective expertise and objective reference.

Declaring a category of RPG questions off topic usually only happens after there's clear evidence of ongoing systematic major breakdowns & malfunctions inherent to handling topic on our site. Whilst these questions have some hitches (like sometimes OPs revise/update the homebrew in the question after feedback and expect answers to update accordingly, which isn't how we work here) that's nowhere near the kinds of breakdown levels that renders stuff off topic and can be handled well by normal community moderation.

Citation standards on homebrew balance questions

Now, usually, homebrew questions should be supported by experience instead of simply being reformulated brainstorming results. But how could you possibly provide an experience-based answer if the only person who has ever been in contact with the respective homebrew so far is its creator, who also asked the question?

You seem to be referencing our policy on Is homebrew an acceptable answer to a question?, but you're taking it too broadly: it's not meant to be applied to every answer that is in some way dealing with homebrew. The context of this meta policy is that we were running into situations like the following:

Q: How can I speed up combat in D&D 3.5e?

Bad homebrew answer: Here, use this custom super-fast combat system I made up just now in about ten minutes just now. I'm sure it will work great and speed up combat, even though I've never used it nor even seen anything like it and have no evidence it even works at all in practice.

Good homebrew answer: My group was also dissatisfied with combat speeds. We used these techniques/mechanisms/tools/homebrew to speed it up. Here's how it worked out for us in practice, and here's the specific benefits and drawbacks we experienced. Here's how I recommend applying it to your situation.

That policy was written in the context of parsing apart those bad answers from the good answers. It's a narrower application of the Back It Up! principle from Good Subjective, Bad Subjective. Boiling it down, you should ground your answers in objective references, or based on comparable subjective experience by you or someone else that you can cite. In that narrow application, it means we don't want random ideas but actual tried-and-tested solutions that will work. In general it means we want to have some confidence you know what you're talking about and aren't basing your answer entirely off your personal opinion.

The homebrew balance questions I usually see reference back to both personal experience and objective fact. I'd check the homebrew+balance tag intersection as well as your own query: answers to these questions frequently use existing game features as a comparison baseline (e.g. comparing a homebrew cantrip to other spells) or point to the game's own balance guidelines (e.g. the loot guidelines, monster construction guidelines, or encounter difficulty guidelines). One of the questions you cite (Is this homebrew entropic spear balanced?) received relatively mediocre answers, but they still reference standard wealth-by-level guidelines and draw comparison to other similar items.

In your answer to Does not having flight nerf a warlock's familiar?, you're answering based on both objective comparison to other familiars and based on subjective experience about how familiars work out in gameplay. This is the kind of subjective expertise we like to see people draw upon when they back up their questions: you're demonstrating an understanding of how things play out at the table, and using that to inform the advice you're providing, so as to give us some confidence your advice can work out. (The homebrew policy I described above is a narrower application of this expectation.) Similar expertise gets drawn upon when examining a pokémon master class's balance: an experienced player can draw upon comparable personal experience to advise on the homebrew in question.

General helpfulness now & in the future

Another argument against such questions is that they're not really going to help future readers - which doesn't automatically disqualify a question, but hugely limits is usefulness (after all, providing signposts is one of the main upsides of duplicates).

Our primary mission is to answer the question someone has and solve their specific problem. Helpful guidance for future readers is our secondary goal that's produced as a side-effect that we also value and curate. Our homebrew balance questions meet both goals: (1) we give the person feedback on the relative balance implications of the homebrew they've brought up, and (2) we provide a source of homebrew balance advice that may apply to future readers in comparable situations. (They may be few and far between, but we don't aim solely for those questions that maximize impact and relevance.)

All seems OK here.

So, homebrew balance questions are relevant to our expertise and are answered drawing upon that expertise in line with our objective and subjective citation guidelines. There are some mediocre answers and some bumps we run into from time to time, but that's par for the course. There's no major breakdown or mismatch warranting off-topic-ness. We can assist people effectively, and over time build up a body of relevant advice about homebrew balance. Everything seems OK enough to leave them on topic.