Because RestController.php is a file and not one of your "mappings", the last rule will unconditionally serve a 403.
You need to either, make an exception for RestController.php and/or use the END flag (new in Apache 2.4), instead of L, to prevent all further processing of the mod_rewrite directives. (The L flag only stops the current round of processing. In a directory context, like .htaccess, the rewritten URL is passed to the next phase of processing and the last rule is then processed during the second phase.)
You also don't need an explicit rule to serve the 404, since that will happen by default. You only need a rule when the request maps to a file (or directory) in order to serve a 403.
For example, with an exception for RestController.php in the last rule block:
:
# catch all non-matched requests and throw 404 or 403 accordingly
RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule !^RestController\.php$ - [R=403]
# A 404 will naturally be served otherwise...
You don't need the L flag when using a non-3xx status with the R flag.
You don't necessarily need to check for a directory (2nd condition), unless you happen to have a DirectoryIndex document (eg. index.php or index.html) in that directory (but why would you?). A 403 is served automatically by mod_autoindex when requesting a directory with no DirectoryIndex document.
Alternatively, or as well as, use the END flag in the preceding rules:
# REST mappings
RewriteRule ^json$ RestController.php?key=json [END,NC,QSA]
RewriteRule ^file/(.*) RestController.php?key=file&id=$1 [END,NC,QSA]
RewriteRule ^actions$ RestController.php?key=actions&id=none [END,NC,QSA]
RewriteRule ^actions/(.*) RestController.php?key=actions&id=$1 [END,NC,QSA]
# catch all non-matched requests and throw 404 or 403 accordingly
RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^ - [R=403]
# A 404 will naturally be served otherwise...
The NC flag should not be necessary here. Note also that I removed the $ in ^file/(.*)$ since it's not required (regex is greedy by default).
Aside:
# REST mappings
RewriteRule ^json$ RestController.php?key=json [END,NC,QSA]
RewriteRule ^file/(.*) RestController.php?key=file&id=$1 [END,NC,QSA]
RewriteRule ^actions$ RestController.php?key=actions&id=none [END,NC,QSA]
RewriteRule ^actions/(.*) RestController.php?key=actions&id=$1 [END,NC,QSA]
Your "mappings" could be reduced to a single rule if you standardise the endpoints, ie. Always a key and id parameter and id could be empty (not "none"), which you would need to validate for anyway (both /actions and /actions/ are valid requests according to your rules, but result in different targets). You then validate the target in your PHP script.
For example:
RewriteRule ^(json|file|actions)(?:/(.*))?$ RestController.php?key=$1&id=$2 [END,NC,QSA]
This does mean that /json/<foo> and /file would be successfully rewritten to RestController.php (which would otherwise be ignored by your original rules), but these requests should be handled by your REST API validation anyway. Perhaps .* should really be .+.
