We generally want encryption output to look random, so I doubt that any encryption function like you want exists.
But I think you can achieve your goal with a two-stage process.
- Encrypt your data using AES, using an appropriate mode such as CBC or GCM. This stage is responsible for security.
- Convert each AES ciphertext block to the form you need. Here you can use any algorithm that converts a 128-bit integer to a vector format that fits your goal. It would probably be convenient if this can handle 128-bit blocks, but if that is too large, you can break the block into (say) four 32-bit blocks and handle each of the smaller blocks separately. The recipient would then convert each vector back to 32-bit blocks, and concatenate four successive 32-bit blocks to recreate the original 128-bit AES ciphertext block, and feed that block into the appropriate AES decryption algorithm. This stage only converts the data to the desired format, and is not intended to add any security.
As an example of a stage 2 algorithm, suppose we want our output to be the value y in the polynomial y = x2 + 3. So we take our 128-bit AES ciphertext block c as a integer, calculate o = c2 + 3, then use o as our final output. (I suspect this example is not quite what you are aiming for, but hopefully points you in an appropriate direction).
Note: Make sure your conversion algorithm is invertible (so any incoming o can be uniquely converted back to the original c) or decryption will not work properly.