I have an mlflow server that is hosted on aws, that uses s3 in the background for storage. I am trying to run a lambda-function that runs a python script which accesses the mlflow server, loads a pytorch model, makes predictions and finishes. The code where my problems arise is:
with open('./data/api_keys.json', 'r') as f:
API_KEYS = json.load(f)
os.environ['AWS_DEFAULT_REGION'] = API_KEYS['AWS_DEFAULT_REGION']
os.environ["AWS_ACCESS_KEY_ID"] = API_KEYS['AWS_ACCESS_KEY_ID']
os.environ["AWS_SECRET_ACCESS_KEY"] = API_KEYS['AWS_SECRET_ACCESS_KEY']
mlflow.set_tracking_uri(MLFLOW_TRACKING_URI)
model = mlflow.pytorch.load_model(MODEL_URI) # <---- error thrown here
The traceback from cloudwatch is:
File "/app/production_script.py", line 26, in <module>
model = mlflow.pytorch.load_model(MODEL_URI)
File "/usr/local/lib/python3.9/site-packages/mlflow/pytorch/__init__.py", line 693, in load_model
local_model_path = _download_artifact_from_uri(artifact_uri=model_uri)
File "/usr/local/lib/python3.9/site-packages/mlflow/tracking/artifact_utils.py", line 95, in _download_artifact_from_uri
return get_artifact_repository(artifact_uri=root_uri).download_artifacts(
File "/usr/local/lib/python3.9/site-packages/mlflow/store/artifact/artifact_repo.py", line 179, in download_artifacts
if self._is_directory(artifact_path):
File "/usr/local/lib/python3.9/site-packages/mlflow/store/artifact/artifact_repo.py", line 61, in _is_directory
listing = self.list_artifacts(artifact_path)
File "/usr/local/lib/python3.9/site-packages/mlflow/store/artifact/s3_artifact_repo.py", line 121, in list_artifacts
for result in results:
File "/usr/local/lib/python3.9/site-packages/botocore/paginate.py", line 269, in __iter__
response = self._make_request(current_kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/paginate.py", line 357, in _make_request
return self._method(**current_kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 530, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 960, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidToken) when calling the ListObjectsV2 operation: The provided token is malformed or otherwise invalid.
As per the docs, ListObjectsV2
Returns some or all (up to 1,000) of the objects in a bucket with each request.
To use this operation, you must have READ access to the bucket.
To use this action in an AWS Identity and Access Management (IAM) policy, you must have >permissions to perform the s3:ListBucket action.
In my policy that I've defined for this lambda-function, I have s3:ListBucket permission for the bucket that mlflow is hosted on.
Any ideas on how to fix this are welcome.
