In section 3.2 "Modes of use of sponge functions" of Cryptographic Sponge Functions (CSF for short for the purpose of this answer), the authors listed several "modes of operations" for different purposes, with stream cipher applications being one of them. Here's the table:
Functionality |
Expression |
Input |
Output |
$n$-bit hash function |
$h = H(M)$ |
$M$ |
$\lfloor Z \rfloor _n$ |
$n$-bit randomized hash function |
$h = H_R(M)$ |
$R|M$ |
$\lfloor Z \rfloor _n$ |
$n$-bit hash function instance differentiation |
$h = H_D(M)$ |
$D|M$ |
$\lfloor Z \rfloor _n$ |
Slow $n$-bit one-way function |
$h = H_{\text{slow}}(M)$ |
$M|0^N$ |
$\lfloor Z \rfloor _n$ |
$n$-bit MAC function |
$T = MAC(K,[IV,]M)$ |
$K|IV|M$ |
$\lfloor Z \rfloor _n$ |
Random-access stream cipher ($n$-bit block) |
$z_i = F(K,IV,i)$ |
$K|IV|i$ |
$\lfloor Z \rfloor _n$ |
Stream cipher |
$Z=F(K,IV)$ |
$K|IV$ |
as is |
Deterministic random bit generator (DRBG) |
$z=DRBG(seed)$ |
seed |
as is |
Mask generating and key derivation function |
$mask=F(seed,l)$ |
seed |
$\lfloor Z \rfloor _l$ |
Is there any literature on using SHAKE256_XOF as a cipher?
Well, the paper has you covered.
How does this method compare to AES256 in terms of security?
This method has equal strength in terms of brutal force. a SHAKE stream cipher will be able to have longer periods than AES-256 either counter mode (CTR) or cipher feedback mode (CFB).
What are there weaknesses or disadvantages of this cipher? Obviously, the use of SHA3-512 to make it authenticated means it can't be efficiently encrypted in one pass and decrypting requires calculating the SHA3-512 digest to authenticate (I am learning how authenticated encryption works so suggestions and feedback are welcomed).
Actually, if you look at "duplex" modes (which the CSF paper also covers), AEADs are actually easier to implement in one pass.
Does it provide any possible advantage over AES256?
I've mentioned 2 (period length, 1-pass). It also has better parameter scalability than fixed-width block-cipher, so I'm counting this as the 3rd.
AES256 has 128-bit resistance against theoretical quantum attacks. Would this cipher maintain the min(key_length/2, 256)
bits of resistance to theoretical quantum attacks that SHAKE256_XOF has.
Probably, but I'm not too good at maths.