While you can create your own hash function, symmetric cipher, public key encryption scheme, etc. such that no one would know its design, it is probably a bad idea to rely on obscurity (as @kelalaka commented) to obtain security.
In the 19th century, Auguste Kerckhoffs stated a principle we call as "Kerckhoffs's principle":
a cryptosystem should be secure even if everything about the system, except the key, is public knowledge.
Additionally, Shannon's Maxim was also formulated in the 20th century and it states that:
one ought to design systems under the assumption that the enemy will immediately gain full familiarity with them
Or in other words - when you design a system for security, you must assume that the attacker knows how the system operates.
In your example of a proprietary hash function, the attacker might somehow get access to the software or hardware that computes such a hash function, and using crypto-analysis or backwards engineering it will manage to find vulnerabilities such as collision attacks or pre-image recovery techniques.
Another reason not to rely on proprietary or secret techniques is consumer trust: Why would a consumers trust that you keep their (hashed) password secure if you admit that the security comes from not knowing the implementation?
This is why in industry, the cryptographic primitives (hash functions, ciphers, signature schemes, etc.) are often only adopted after significant scrutiny from the scientific community. Then, RFCs and standards are being drafted so that vendors and service providers can quantify their security by saying which standard they comply to.
