The answer to this question on SO explains what you're looking at. The picture you shared shows that your file begins with an escape character (the "E", "S" and "C" on the diagonal) followed by "Lup" in ASCII. This is the 4 byte signature placed at the top of a Lua bytecode file to make it easier to identify.
Bytecode is not source code, but code translated (programmers say "compiled") into a more efficient form for interpretation. The computer must do work to execute Lua code you write: compiling does a bunch of that work - the work that's going to be the same every time your code is run - and saves the result in a compact form. That form is designed for the computer, not people, so it's not easy to read or edit. There are almost as many different bytecodes as there are interpreted programming languages, but instructions are often a single byte (compact and efficient), which is where the name comes from.
You need to find the Lua source code so you can make changes, then recompile to create new bytecode. Really, any text editor should be fine for editing the source once you've got it - there are tons of text editors, IDE's etc that should do the job (even the one in your picture).
I recommend spending some time on the question linked to above - I think it will be worth the investment!