Yes, you can use that. By definition an attacker should not be able to come up with a signature over any specific message without the private key. So if that's the case then a signature can indeed be used as a symmetric key, and I know of one instance where it was used like that (or at least as a secret, rather than a secret key).
However, you should be careful when doing so. Some signature algorithms are indeterministic, and that could complicate things. In the worst case it could make the signature algorithm entirely insecure. At best you might have to alter the signature generation routine to inject a seed or random value. It would also require you to store that the seed or random value, and that might go against your use case requirements. Note that if anything about the random number generation is changed - the algorithm, the extraction mechanism or the integer extraction - then you will likely get the wrong key (and yes, I've seen this happen).
However, if you'd use a deterministic version of RSA (PKCS#1 v1.5 padding) or deterministic ECDSA then you should be alright.
You are right to use a KDF after the data however. A symmetric key should extract all the randomness within the signature to be secure (possibly followed by key expansion to get to the right size, but that's commonly not required).
scrypt
however is generally used on passwords. That's fine, but you probably don't need the salt or the work factor to stretch the key (as the amount of randomness in the signature should be high enough anyway). So it is fine if scrypt
is required, but normally you'd rather use a KBKDF such as HKDF or just HKDF-Extract.
If you have access to the private key value you might as well use the KDF on that.
I'd definitely recommend that you'd use a signature generation algorithm that has at least the same strength as the symmetric key. Same goes for the KDF function of course; you can find details at keylength.com.
Furthermore, you might want to keep in mind that symmetric cryptography is hard to attack with a full fledged quantum computer, while asymmetric algorithms are not (although this scheme may be harder to attack than a generic signature generation scheme, depending on how it is used).
Yet another implementation note: signature generation functions don't expect that you keep the signature secret. For instance, a hardware module might be able to keep a normally derived symmetric key inside of the HSM. On the other hand, it might export a signature without fuss as a signature value is supposed to be public.
Side channel attacks on the signature value might also be possible, it is hard to tell in advance. So although this scheme should be theoretically sound, it is still highly inadvisable to do so unless there is no other way.