What [is] mainstream cryptography?
I would define mainstream cryptography as "the kinds of papers that get published at IACR venues" (Crypto, Eurocrypt, Asiacrypt, TCC, PKC, CHES, FSE, Journal of Cryptology).
The IACR is the main professional society for cryptography research.
BTW, I'm not trying to say that these are the only legitimate venues for cryptographic research.
There are many other excellent venues, but the cryptography papers published there would also be largely suitable for IACR venues too.
Is its definition subjective?
The norms and conventions of any scientific community are established by the humans that practice that science.
Humans suffer from all sorts of biases.
There is always an element of arbitrariness to what is considered mainstream practice and what isn't.
I seem to recall that Mihir Bellare's distinguished lecture discusses some of these elements in the context of cryptography.
For example, the choice between asymptotic vs concrete statements of security is arbitrary and socially constructed by the community -- so is the choice of which problems are popular at any given time.
Bellare references Kuhn's Structure of Scientific Revolutions as a resource that outlines the concepts more generally.
Is symmetric cryptography within the mainstream cryptography?
Yes, there are proposals for new symmetric cryptography and attacks on existing symmetric cryptography, at almost every IACR conference.
Additionally, I would guess that the vast majority of cryptography papers use at least some symmetric-key cryptographic building blocks somewhere.
Within symmetric cryptography, there are mainstream approaches and non-mainstream approaches.
For example, basing a hash function on a wide public permutation is mainstream -- most new hash functions use this design paradigm.
Justifying your designs by showing resistance to linear/differential cryptanalysis is the mainstream expectation.
Basing a hash function on "chaotic systems" is not mainstream, and I cannot recall seeing this kind of work published at IACR venues.
Are there good cryptography out of the mainstream?
Almost certainly.
If so, how can we judge chaos-based cryptography being out of the mainstream?
It is reasonable to be skeptical if someone dismisses an area simply for being outside the mainstream.
You should ask why it is outside of the mainstream.
In the case of chaos-based cryptography, many such reasons have been given here on stackexchange.
