Score:0

installing packages in AWS EMR

cn flag

I'm trying to install Google Tink in AWS EMR 5.28.0 without much luck. It looks like the AWS EMR image is rather strange in nature. Any ideas?

sc.install_pypi_package("tink")

error: Could not find bazel executable. Please install bazel to compile the Tink Python package.

 sudo yum install -y bazel3

Requires: java-11-openjdk-devel

sudo yum install java-11-openjdk-devel

No package java-11-openjdk-devel available.

 sudo yum install amazon-linux-extras

No package amazon-linux-extras available.

Michael Hampton avatar
cz flag
What repositories do you have enabled?
Koenig Lear avatar
cn flag
@MichaelHampton I don't know how to enable repositories?
Michael Hampton avatar
cz flag
You are new to Linux? Start with the [documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-linux-ami-basics.html#package-repository).
Koenig Lear avatar
cn flag
@MichaelHampton I have enabled sudo yum-config-manager --enable epel. No change, I would have thought jdk is a pretty common package.
Michael Hampton avatar
cz flag
What about all the other repos?
Koenig Lear avatar
cn flag
@MichaelHampton yes how do you install them? I can't see anything in the aws doc
Michael Hampton avatar
cz flag
Now you know about yum-config-manager. You can use that to inspect the state of the system and make any necessary changes.
Koenig Lear avatar
cn flag
@MichaelHampton that's not a useful comment
Michael Hampton avatar
cz flag
I am not here to read documentation for you. You should have already read about yum-config-manager and learned what you can do with it.
Score:0
cn flag

As it turns out the Linux AMI (The base for AWS EMR 5.28.0) is a pain to work with. None of the available repositories have Java, Bezel, much less Tinker readily available (compare to Macos, Ubuntu, or even ECR Python image where it's a breeze)

To get Java 11jdk you need to go to Oracle and look for Linux x64 RPM Package. Unfortunately you need to sign in to their site so wget/curl is not possible.

To install you need to do:

 sudo yum localinstall jdk-11.0.12_linux-x64_bin.rpm

For Bazel you need to get the package from Github https://github.com/bazelbuild/. Pick the latest and install

./bazel-4.1.0-installer-linux-x86_64.sh 

Then pip works fine

sc.install_pypi_package("tink")

Output:

Successfully built tink
Installing collected packages: protobuf, absl-py, tink
Successfully installed absl-py-0.13.0 protobuf-3.17.3 tink-1.6.1
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.