How to achieve encryption at single source and decryption at multiple locations using asymmetric scheme?

I need an encryption scheme where a single source/location can encrypt using their private key and while anyone with access to the corresponding public key can decrypt it. It has to be asymmetric to avoid any one else deriving that private key and pretending to become that source. For example, government issues an encrypted document that any one authorized with the issued public key can decrypt it. But no one else should be able to mimic the government issued document. So confidentiality and integrity are both solved.

If GPG can address it, how secure is it and is it possible to use HSM or is it supported pcks.

The question (version 2) uses incorrect terminology:

• One does not "encrypt using their private key". One signs using their private key.
• Then "anyone with access to the corresponding public key can" verify (not "decrypt") the signature against what has been signed (if that's available).

In the following I'm assuming the above correction is made to the question, up to this:

So confidentiality and integrity are both solved.

No. That's a serious misunderstanding. Signature gives integrity and publicly verifiable authenticity, but not confidentiality. Signature is not encryption, if only because encryption implies that only those with a secret can understand the original message, and the public key is assumed available to all, thus not secret. Besides, the most usual forms of signature (with appendix, as in RSA-SSA-PSS, RSA-SSA-PKCS1-v1_5, DSA, ECDSA, EdDSA…) assume (save for other mechanism) that the message is sent in clear to the party that does the verification, which goes against the goal of confidentiality.

It's possible to have both confidentiality and integrity (understood as publicly verifiable authenticity), but that requires encryption for the former goal, and signature for the later. And the same asymmetric key pair can not be used for both goals (since the the private key used for signature must be available to the party that signs, and would need to be available to the party that deciphers if the same key pair was used for encryption).

If GPG can address it, how secure is it and is it possible to use HSM or is it supported pkcs

Yes GPG can address both signature and encryption (for the later: both symmetric and asymmetric). It supports many secure algorithms for both goals.

Yes GPG can use an HSM, in particular to generate and hold the signature private key, and sign. In an ideal world, that would just work thru the PKCS#11 mechanism of standard HSMs. In practice it does not and there is a GPG adapter for each brand of HSM. Typically, googling GPG HSM (brand-of-HSM) will return the appropriate documentation. The support is somewhat better for Smart Cards.

GPG is far from the only tool for this. And the necessary cryptography is built into some readers for some file formats, in particular PDF.

It's possible to perform signature and encryption thru the PKCS#11 mechanism of HSMs, and that's often done in e-government. But that requires carefully defined specification, and a good working knowledge of cryptography.

_{Note: PKCS is a family of standards, like ISO. Without a number or context it does not specify any particular thing.}