Is finding an encryption (or hash) algorithm science, or is it art?

I'm not into cryptography, but I read a bit about it. I wonder:

AFAIK all encryption (decryption) and hashing routines are built using some very primitive functions (let's name them "atoms" for now) with specific properties.

As the set of atoms is probably rather small, a suitable algorithm consists of "combining" such "atoms" to (let's name them) "molecules" in a way that produces "suitable" molecules. Maybe there are also additional "glue" functions that are used to combine "atoms" to "molecules", but for simplicity, let's assume there is just one set of "atoms".

Now I wonder: Is building arbitrarily large "suitable molecules" a science, or is it an art?

I mean: If it's science, is there an algorithm to build arbitrarily large "suitable molecules" from a given set of "atoms" (and "glue")? Such an algorithm would also find out if it's not possible, of course.

However, if it's an art, there is some genius needed who provides a "molecule" (using "atoms" and "glue"), claiming (until proven otherwise) that it has the properties demanded.

Currently, it seems to be an art; otherwise, it would be so easy to construct better and safe algorithms, right?

(I'm not sure whether I should have replaced "science" with "mathematics" in this question, because "science" may also include some "intuitive" components, that mathematical solutions do not need)

I also wonder whether the strength of a cryptographic algorithm has to do with the fact (I think) that there is no reasonable (means: not "trial and error") algorithm to build arbitrarily strong algorithms automatically. Would such an algorithm (if it existed) mean that breaking such functions is just as easy? If not, what part of the maths make the difference?

Without getting into the weeds of "is modern cryptography art or science", one can pretty conclusively say that classical cryptography was an art, and was not particularly secure. The history of cryptosystems people might still consider secure goes back to the 1930s at the earliest, despite hundreds of years of previous military interest in cryptography.

This is to say that the design of cryptosystems is definitely more scientific than the previous design of cryptosystems, and our cryptosystems now are more secure.

Is building arbitrarily large "suitable molecules"...

This analogy is wrong. In chemistry, large molecules may have new important properties and building large molecules may be desired.

But in cryptography adding more complexity is usually a disadvantage, because it takes more resources without giving any positive effects. In many cases, the cryptography tries to solve a very different problem: How to reach the desired quality with as little resources (CPU, memory) as possible? In your wording it would be how to build as simple molecules as possible.

It is more science than the art. For instance, the strength of RSA and ECC has a solid mathematical base. Also many kinds of attacks require a solid mathematical knowledge in the first place. Even to benefit from side channel leaks, one needs again a solid mathematical knowledge.

The design of cryptographic mechanisms is a science: what makes a cryptographic mechanism good is its resistance to attacks, not its aesthetics.

I mean: If it's science, is there an algorithm to build arbitrarily large "suitable molecules" from a given set of "atoms" (and "glue")? Such an algorithm would also find out if it's not possible, of course.

This does not follow from any modern definition of “science”. Science means that there are rules, not that we know them.

However, if it's an art, there is some genius needed who provides a "molecule" (using "atoms" and "glue"), claiming (until proven otherwise) that it has the properties demanded.

That's one possible sense of the word “art”, but not the usual one. “Art” in the modern sense implies that there's no single right answer. And the security of cryptographic primitives does have answers, we just don't know them for sure.

I think the distinction you're trying to make is between a theoretical science and an experimental science. Cryptography is both (as are physics, biology, etc.). One can design a cryptographic mechanism as a pure theory. But what makes it recognized as good is that other people fail to break it, and that's a form of experimental validation.

It's art in the first place I suggest. Otherwise how can you think of anything new?

Somebody mentioned a comparison with buildings. Sagrada Família was not born of numbers nor abaci. It was born of love for architecture, nature, and religion. Now to be sure it's being completed (any day) by engineers using 3D modelling.

Keccak was not created by an incremental change to SHA2. Some other competitors were though and just re-hashed (get it?) the old Merkle-Damgård construct. It took creative inspiration to think of a spongy cube where bits are mangled, merged and then squeezed out. I also posit that a significant part of it's winning fortune was the imagery in the submission. The piccies are great. They could have been rubbish ASCII graphics like seen in RFCs. Piccies are art. Then some cryptography was applied.

Encryption algorithms used to get more and more complex. Lucifer -> DES -> AES -> post quantum. Then someone had the inspiration to think; what if we made something simpler with more rounds? What if we just add, rotate and xor? That didn't come from a text book. Given Speck's philosophy and 34 rounds, it shouldn't be hard for many people to conceive even simpler architectures given a thousand rounds. It may take some engineers to make that design secure with less than a thousand though.

Richard Wollheim suggests that the above can be squared by the idea that art cannot be defined without consideration of it's particular environment. I believe that to be true. But I also maintain that fundamental ideation is art first, to be followed by technical realisation.