From my point of view, it depends a bit what you're targetting exactly:
Linux distributions like Fedora actually expect for their distribution builds that the software is built from source using the spec file (inside a sealed buildsystem that's also documenting the packages/components used for building) and consider this a good practice, also to gain as much reproducibility as possible.
However, you indeed can just pull pre-built binaries and use them inside a spec file to build a RPM package, shortened example for your scenario:
# […]
Source0: <binary file>
Source1: <config file>
# […]
%prep
%setup -q -c -T
%build
%install
install -D -p -m 0755 %{SOURCE0} $RPM_BUILD_ROOT%{_bindir}/%{name}
install -D -p -m 0644 %{SOURCE1} $RPM_BUILD_ROOT%{_sysconfdir}/%{name}.conf
# […]
While this might work, you should explicitly check whether the dependencies of the resulting RPM package are suitable (rpm -qp --requires <…>.rpm) or if there are missing run-time dependencies. Often there are compile-time options or parameters (e.g. CFLAGS or LDFLAGS software written in C) as well as post-processing scripts (e.g. /usr/lib/rpm/(redhat/)brp-*) while building the RPM package that add distribution specific stuff and/or extract run-time dependencies for you. This may or may not be relevant for you (and some post-processing scripts also might work for pre-built binaries but other do not due to lack of compiler options). While I'm packaging a lot of software, I'm not experienced with Haskell, but when looking to some Fedora packages, there are RPM packages having run-time dependencies (and if there are run-time dependencies these have to be satisfied by RPM packages, because that's how RPM works; and it's not enough to have the matching library somewhere on your filesystem, because RPM will not know about it except it's provided by a RPM package). Thus, building a RPM package from pre-built binaries might lead to a RPM package without proper dependencies which might reveal as run-time errors later.
Regarding the above example: While SourceX: can be given a URL, rpmbuild still expects these files on-disk in the SOURCES directory when building the RPM package (so how they're getting into the SOURCES directory is up to you).
Given you didn't mention which Linux distribution you're targetting, I'm linking Fedora's Haskell Packaging Guidelines
here, which may or may not provide further inspiration. And yes, RPM packages can be cross-distribution, which practically often leads to static binaries inside the RPM packages, due to often different library/package versions on different Linux distributions.
