We had an App Engine's standard environment application running in Python 2.7, and we upgraded it to Python 3.11.
Services/frameworks used by old app:
- Shared Memcache
- Task Queue
- Google's Discovery library (for Big Query)
- webapp2 Framework
Services/frameworks used by new app:
- Flask
- Shared Memcache
- Task Queue
- Storage Write API
- webapp2 was removed since it isn't compatible with Python 3.
We followed this Google's guide to upgrade our application and use the legacy bundled services: https://cloud.google.com/appengine/docs/standard/python3/services/access#python-3
The codebase is simple. It reads a request and stores it in Big Query. It uses the Memcache service to throttle requests and the Task queue to call the "Store to Bigquery" endpoint in the same application.
After the upgrade we saw the following:
Higher latency: from ~50ms to ~250ms
Higher number of instances: from 6 to 50 using the same workload
We tried to run the Google's profiler but the platform couldn't install the package for Python 3.11.
We created and deployed a "hello world" application to discard performance issues with Python 3.11, Flask, Memcache, Taskqueue, and BigQuery streaming insertion.
We found that the BigQuery legacy insertion was causing a latency delay of ~500ms. Then we implemented the Storage Write API but we ended with a latency of ~250ms, which isn't comparable with the previous performance.
Is this an issue with the App Engine platform?
Are there another steps to avoid this kind of delays?
We expect the same or better performance as the previous version.
