Given the subgroup $H=\lbrace 1, \omega, \omega^2, \omega^3, ..., \omega^{n-1}\rbrace$ of order $n$, the Lagrange basis of this subgroup is given by the set of polynomials $\lbrace L_i \rbrace_{i=1}^n$ such that $L_i(w^k) = 1$ if and only if $i=k$ and $0$ otherwise.
The well-known interpolation methods gives $L_i(x) = \frac{\prod_{k \neq i}(x - \omega^k)}{\prod_{k \neq i}(\omega^i - \omega^k)}$.
As per the definitions of the basis, the $L_1$ expression in the question
$$L_1(X) = \frac {(X-\omega)(X-\omega^2)(X-\omega^3)...(X-\omega^{n-1})}{(1-\omega)(1-\omega^2)(1-\omega^3)...(1-\omega^{n-1})}$$
does not compute the correct value and is instead computing $L_n$ ($L_0$ given that $\omega^n = \omega^0 = 1)$.
$L_1(x)$ should instead be $$L_1(X) = \frac {(X-1)(X-\omega^2)(X-\omega^3)...(X-\omega^{n-1})}{(\omega-1)(\omega-\omega^2)(\omega-\omega^3)...(\omega-\omega^{n-1})}.$$
Using the numerical example, it can be verified that $L_1(5) = 12$, which corresponds to $L_1(5) = \frac {4(5^4 -1)}{4(5-4)}$.
This can also be verified using the sagemath snippet below
F = GF(17)
w = F(4)
s = 5
idxs = lambda i: [k for k in range(4) if i != k]
li = lambda i,x: prod([x - w^k for k in idxs(i)])/prod([w^i - w^k for k in idxs(i)])
# Sanity check
[[li(j,w^i) for i in range(4)] for j in range(4)]
# L_i(s)
print([li(i,s) for i in range(4)])
The output is [5, 12, 8, 10]. Which also shows that $5$ the result of $L_n(5)$.
Now, what about the connection between the "normal" Lagrange polynomial and the one defined in the paper? The paper "Fractal: Post-Quantum and Transparent Recursive Proofs from Holography" explained this. At a high level, the vanishing polynomial $Z_H(x)$ for $H$ and the polynomials $L_i$ are quite related.
Indeed, $Z_H$ is almost the same as $Z_H(x)/(x-\omega^i)$. But, not quite, since the latter polynomial does not evaluate to $i$ in $\omega^i$, so we need some normalization.
Consider the derivative of $Z_H(x)$, when evaluated at $w^i$ we get $$Z_H'(w^i) = 1\times\prod_{k \neq i}(\omega^i - \omega^k) + (w^i-w^i)\times\text{stuff} = \prod_{k \neq i}(\omega^i - \omega^k).$$
Consequently, we can see that $$L_i(x) = \frac{1}{Z_H'(\omega^i)} \frac{Z_H(x)}{(x-w^i)}.$$
And $$L_1(x) = \frac{1}{n\omega^{-1}} \frac{x^n - 1}{(x-w)}.$$
