DEV Community

ilija
ilija

Posted on • Edited on

Web3 backend and smart contract development for Python developers Musical NFTs part 17: Deployment time!

#python #django #ethereum #googlecloud

Now we want to have our Django app up and running somewhere live for other people to see and use. In this version of our app we will use Google App engine.

First go to Google cloud and open your free account. Google will allocate some free resources automatically to your account and this will be more then enough for start. But also don't forget to go to billing part and to set some alerts when certain price thresholds are hit. Just to avoid any surprise.

Ones you are there search for App engine and create and select new MusicalNFT project.

Image description

In short what is Google App Engine (GAE)? It is a fully managed, serverless platform for developing and hosting web applications at scale. It has a powerful built-in auto-scaling feature, which automatically allocates more/fewer resources based on demand. GAE natively supports applications written in Python, Node.js, Java, Ruby, C#, Go, and PHP. Alternatively, it provides support for other languages via custom runtimes or Dockerfiles. It has powerful application diagnostics, which you can combine with Cloud Monitoring and Logging to monitor the health and the performance of your app. Additionally, GAE allows your apps to scale to zero, which means that you don't pay anything if no one uses your service.

In attempt to use GAE first we need to install Google Cloud CLI.
The gcloud CLI allows you to create and manage your Google Cloud resources and services. The installation process differs depending on your operating system and processor architecture. Go ahead and follow the official installation guide for your OS and CPU. => gcloud

To verify the installation has been successful, run:

$ gcloud version Google Cloud SDK 415.0.0 bq 2.0.84 core 2023.01.20 gcloud-crc32c 1.0.0 gsutil 5.18
Enter fullscreen mode Exit fullscreen mode

Now comes very important step: configuring Django Project to work with GEA. Till this moment we were working all the time in local dev environment, running all things (Postgres, Stripe, Django server, Celery etc.) on our machine. What we need to do now if we want our app to be live on GEA? First we need to configure our Django project and to tell to GEA that he need to run all those supporting services in the background for our app to be able to run smoothly. That is why let's open our Django settings.py

First thing we need to do is to erase defualt secrete keys. Then we will generate new one and pass to our .env file. And only then import inside our settings.py again.

Let's generate new secrete Django key:

 $py manage.py shell >>>from django.core.management.utils import get_random_secret_key >>>secret_key = get_random_secret_key() >>>print(secret_key) # and you will get some gibberish like this >>>^&p@m*nhn#hg6ujgri2sppxr7t8o^mfp3bnj%1%2f72wcr+kkz # now pass value you get into `.env` file name of varibale `DJANGO_SECRET_KEY`
Enter fullscreen mode Exit fullscreen mode

Why we need secret key? In Django, a secret key plays a vital role in enhancing the security of our application. It helps manage user sessions, protects against Cross-Site Request Forgery (CSRF) attacks, and safeguards your data by generating and verifying cryptographic signatures among other things.

Now our Django settings should look something like this:

 from pathlib import Path import os from urllib.parse import urlparse import environ import io import pyrebase import firebase_admin from firebase_admin import credentials from google.cloud import secretmanager from google.oauth2 import service_account # Build paths inside the project like this: BASE_DIR / 'subdir'. BASE_DIR = Path(__file__).resolve().parent.parent env = environ.Env(DEBUG=(bool, False)) env_file = os.path.join(BASE_DIR, ".env") if os.path.isfile(env_file): # read a local .env file env.read_env(env_file) elif os.environ.get("GOOGLE_CLOUD_PROJECT", None): # pull .env file from Secret Manager project_id = os.environ.get("GOOGLE_CLOUD_PROJECT") client = secretmanager.SecretManagerServiceClient() settings_name = os.environ.get("SETTINGS_NAME", "django_setting_two") name = f"projects/{project_id}/secrets/{settings_name}/versions/latest" payload = client.access_secret_version(name=name).payload.data.decode("UTF-8") env.read_env(io.StringIO(payload)) else: raise Exception("No local .env or GOOGLE_CLOUD_PROJECT detected. No secrets found.") SECRET_KEY = env("SECRET_KEY") DEBUG = env("DEBUG")
Enter fullscreen mode Exit fullscreen mode

Now set ALLOWED_HOSTS and CSRF_TRUSTED_ORIGINS, we can use the following code snippet from the GAE docs:

 APPENGINE_URL = env("APPENGINE_URL", default=None) if APPENGINE_URL: # ensure a scheme is present in the URL before it's processed. if not urlparse(APPENGINE_URL).scheme: APPENGINE_URL = f"https://{APPENGINE_URL}" ALLOWED_HOSTS = [ urlparse(APPENGINE_URL).netloc, APPENGINE_URL, "localhost", "127.0.0.1", ] CSRF_TRUSTED_ORIGINS = [APPENGINE_URL] # SECURE_SSL_REDIRECT = True else: ALLOWED_HOSTS = ["*"]
Enter fullscreen mode Exit fullscreen mode

This code fetches APPENGINE_URL from the environment (later we will add to our .env) and automatically configures ALLOWED_HOSTS and CSRF_TRUSTED_ORIGINS. Additionally, it enables SECURE_SSL_REDIRECT to enforce HTTPS.

Final version .env file should look something like this (we still don't have all values but just for information)

STRIPE_SECRET_KEY=sk_test_xxxxxxxxx DENIS_PASS=xxxxxxx ETHEREUM_NETWORK=maticmum INFURA_PROVIDER=https://polygon-mumbai.infura.io/v3/xxxxxx SIGNER_PRIVATE_KEY=xxxxxxx MUSIC_NFT_ADDRESS=0x1D33a553541606E98c74a61D1B8d9fff9E0fa138 STRIPE_ENDPOINT=whsec_GExxxxxxx OWNER=0x273f4FCa831A7e154f8f979e1B06F4491Eb508B6 DJANGO_SECRET_KEY='^&p@m*nhn#hg6ujgri2sppxr7t8o^mfp3bnj%1%2f72wcr+kkz' DATABASE_URL=postgres://name:password!@localhost/musicalnft # DATABASE_URL=postgres://name:password!@//cloudsql/musicnft-405811:europe-west1:musicnft-instance/musicalnft GS_BUCKET_NAME=musicnft-bucket APPENGINE_URL=https://musicnft-405811.ew.r.appspot.com/
Enter fullscreen mode Exit fullscreen mode

later on we will populate all those values.

Don't forget to add the import at the top of the settings.py:

 from urllib.parse import urlparse
Enter fullscreen mode Exit fullscreen mode

To use Postgres instead of SQLite, we first need to install the database adapter.

 $pip install psycopg2-binary==2.9.5 $pip freeze > requirements.txt
Enter fullscreen mode Exit fullscreen mode

To utilize DATABASE_URL with Django, we can use django-environ's db() method like so:

 #settings.py DATABASES = {'default': env.db()}
Enter fullscreen mode Exit fullscreen mode

Now create DATABASE_URL in .env and pass some random value, later on we will set up this value properly. (in general format of this value will go as fallow:

postgres://USER:PASSWORD@//cloudsql/PROJECT_ID:REGION:INSTANCE_NAME/DATABASE_NAME)
Enter fullscreen mode Exit fullscreen mode

What we want to do is to replace Django dev server (what is not recommended for any kind of production environment) with Gunicorn. But first what is Gunicorn and why we should want to replace Django default server? The Django development server is lightweight and easy to use, but it's not designed/reccomndent to handle production traffic. It's meant for use during development only. On the other hand, Gunicorn is a WSGI HTTP server that's designed for production use. It's robust, efficient, and can handle multiple requests simultaneously, which is crucial for a production environment.

 $pip install gunicorn==20.1.0 $pip freeze > requirements.txt
Enter fullscreen mode Exit fullscreen mode

Now very important app.yaml file. Google App Engine's app.yaml config file is used to configure your web application's runtime environment. The app.yaml file contains information such as the runtime, URL handlers, and environment variables.

Start by creating a new file called app.yaml in the project root with the following contents:

 # app.yaml runtime: python39 env: standard entrypoint: gunicorn -b :$PORT musical_nft.wsgi:application handlers: - url: /.* script: auto runtime_config: python_version: 3
Enter fullscreen mode Exit fullscreen mode

We defined the entrypoint command that starts the WSGI server.
There are two options for env: standard and flexible. We picked standard since it is easier to get up and running, is appropriate for smaller apps, and supports Python 3.9 out of the box.
Lastly, handlers define how different URLs are routed. We'll define handlers for static and media files later in the tutorial.

Now let's define .gcloudignore. A .gcloudignore file allows you to specify the files you don't want to upload to GAE when deploying an application. It works similarly to a .gitignore file.

Go ahead and create a .gcloudignore file in the project root with the following contents:

 # .gcloudignore .gcloudignore # Ignore local .env file .env # If you would like to upload your .git directory, .gitignore file, or files # from your .gitignore file, remove the corresponding line # below: .git .gitignore # Python pycache: __pycache__/ # Ignore collected static and media files mediafiles/ staticfiles/ # Ignore the local DB db.sqlite3 # Ignored by the build system /setup.cfg venv/ # Ignore IDE files .idea/ README_DEV.md celerybeat-schedule node_modules photos smart-contracts
Enter fullscreen mode Exit fullscreen mode

Go ahead and initialize the gcloud CLI if you haven't already:

 $ gcloud init
Enter fullscreen mode Exit fullscreen mode

gcloud CLI will ask you about mail and project you would like to associate with this project. It will offer to you all avaliable projects. Choose the one we created (MusciNFT) and gcloud will automatically set all the rest. What means from this point on when ever you deploy your Django app to Google App Engine it will be avalible under that project name.

To create an App Engine app go to your project root and run:

$gcloud app create # you will get something like this You are creating an app for project [musicnft-405811]. WARNING: Creating an App Engine application for a project is irreversible and the region cannot be changed. More information about regions is at <https://cloud.google.com/appengine/docs/locations>. Please choose the region where you want your App Engine application located: [1] asia-east1 (supports standard and flexible) [2] asia-east2 (supports standard and flexible and search_api) [3] asia-northeast1 (supports standard and flexible and search_api) [4] asia-northeast2 (supports standard and flexible and search_api) [5] asia-northeast3 (supports standard and flexible and search_api) [6] asia-south1 (supports standard and flexible and search_api) [7] asia-southeast1 (supports standard and flexible) [8] asia-southeast2 (supports standard and flexible and search_api) [9] australia-southeast1 (supports standard and flexible and search_api) [10] europe-central2 (supports standard and flexible) [11] europe-west (supports standard and flexible and search_api) [12] europe-west2 (supports standard and flexible and search_api) [13] europe-west3 (supports standard and flexible and search_api) [14] europe-west6 (supports standard and flexible and search_api) [15] northamerica-northeast1 (supports standard and flexible and search_api) [16] southamerica-east1 (supports standard and flexible and search_api) [17] us-central (supports standard and flexible and search_api) [18] us-east1 (supports standard and flexible and search_api) [19] us-east4 (supports standard and flexible and search_api) [20] us-west1 (supports standard and flexible) [21] us-west2 (supports standard and flexible and search_api) [22] us-west3 (supports standard and flexible and search_api) [23] us-west4 (supports standard and flexible and search_api) [24] cancel Please enter your numeric choice: 11 Creating App Engine application in project [musicnft-405811] and region [europe-west]....done. Success! The app is now created. Please use `gcloud app deploy` to deploy your first app.
Enter fullscreen mode Exit fullscreen mode

Ok, now we need to set up our Postgres database inside Cloud SQL dashboard: https://console.cloud.google.com/sql/

Ones you are there create new instance and choose PostgresSQL (enable Compute Engine API if needed).

Now pass following values:

 Instance ID: musicnft-instance Password: Enter a custom password or generate it Database version: PostgreSQL 14 Configuration: Up to you Region: The same region as your app Zonal availability: Up to you
Enter fullscreen mode Exit fullscreen mode

You might also need to enable "Compute Engine API" to create a SQL instance.

Once the database has been provisioned, you should get redirected to the database details. Take note of the "Connection name".

Go ahead and enable the Cloud SQL Admin API by searching for "Cloud SQL Admin API" and clicking "Enable". We'll need this enabled to test the database connection. Here is a link (it will route you to your project): https://console.cloud.google.com/marketplace/product/google/sqladmin.googleapis.com

To test the database connection and migrate the database we'll use Cloud SQL Auth proxy. The Cloud SQL Auth proxy provides secure access to your Cloud SQL instance without the need for authorized networks or for configuring SSL.

First, authenticate and acquire credentials for the API:

$gcloud auth application-default login
Enter fullscreen mode Exit fullscreen mode

Next, download Cloud SQL Auth Proxy and make it executable:

 wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy --2023-11-21 00:27:00-- https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 Resolving dl.google.com (dl.google.com)... 172.217.20.78, 2a00:1450:4017:800::200e Connecting to dl.google.com (dl.google.com)|172.217.20.78|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 19239740 (18M) [application/octet-stream] Saving to: ‘cloud_sql_proxy’ cloud_sql_proxy 100%[===============================================>] 18.35M 1.22MB/s in 13s 2023-11-21 00:27:13 (1.45 MB/s) - ‘cloud_sql_proxy’ saved [19239740/19239740] $chmod +x cloud_sql_proxy
Enter fullscreen mode Exit fullscreen mode

After the installation is complete open a new terminal window and start the proxy with your connection details like so:

 $./cloud_sql_proxy -instances="PROJECT_ID:REGION:INSTANCE_NAME"=tcp:5432
Enter fullscreen mode Exit fullscreen mode

Where basicaly PROJECT_ID:REGION:INSTANCE_NAME connection name you saved first time you created database

And you should see something like this:

 2023/11/21 00:31:03 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here. 2023/11/21 00:31:04 Listening on 127.0.0.1:5432 for musicnft-405811:europe-west1:musicnft-instance 2023/11/21 00:31:04 Ready for new connections 2023/11/21 00:31:05 Generated RSA key in 135.901349ms
Enter fullscreen mode Exit fullscreen mode

You can now connect to localhost:5432 the same way you would if you had Postgres running on your local machine.

Since GAE doesn't allow us to execute commands on the server, we'll have to migrate the database from our local machine.

Inside our .env file in the project root, with the required environment variables:

DATABASE_URL=postgres://DB_USER:DB_PASS@localhost/DB_NAME # Example `DATABASE_URL`: # DATABASE_URL=postgres://django-images:password@localhost/mydb
Enter fullscreen mode Exit fullscreen mode

Now let's make migrations

 $python manage.py migrate
Enter fullscreen mode Exit fullscreen mode

Create superuser

 $python manage.py createsuperuser
Enter fullscreen mode Exit fullscreen mode

Ones you done with superuser you can move to secret manager. We used to have local .env file when we worked in local dev context. But because we are migrating now whole app into Google App Engine, we need to make our .env varibales present in cloud. And for this we will use Google secret manager.

Navigate to the Secret Manager dashboard (https://console.cloud.google.com/security/secret-manager?project=musicnft-405811) and enable the API if you haven't already. Next, create a secret named django_settings with the following content:

 DJANGO_SECRET_KEY='^&p@m*nhn#hg6ujgri2sppxr7t8o^mfp3bnj%1%2f72wcr+kkz' # local development # DATABASE_URL=postgres://ilija:Meripseli1986!@localhost/musicalnft # in cloud DATABASE_URL=postgres://ilija:Meripseli1986!@//cloudsql/musicnft-405811:europe-west1:musicnft-instance/musicalnft GS_BUCKET_NAME=django-music-nft # Example `DATABASE_URL`: DATABASE_URL=postgres://DB_USER:DB_PASS@//cloudsql/PROJECT_ID:REGION:INSTANCE_NAME/DB_NAME # postgres://django-images:password@//cloudsql/indigo-35:europe-west3:mydb-instance/mydb GS_BUCKET_NAME=django-images-bucket
Enter fullscreen mode Exit fullscreen mode

Make sure to change DATABASE_URL accordingly. PROJECT_ID:REGION:INSTANCE_NAME equals your database connection details.

You don't have to worry about GS_BUCKET_NAME. This is just the name of a bucket we're going to create and use later.

Pip installl google-cloud-secret-manager==2.15.1

 $pip install google-cloud-secret-manager==2.15.1 $pip freeze > requirements.txt
Enter fullscreen mode Exit fullscreen mode

To load the environment variables from Secret Manager we can use the following official code snippet:

 from pathlib import Path import os import environ from urllib.parse import urlparse import io # new from google.cloud import secretmanager # new # Build paths inside the project like this: BASE_DIR / 'subdir'. BASE_DIR = Path(__file__).resolve().parent.parent env = environ.Env(DEBUG=(bool, False)) env_file = os.path.join(BASE_DIR, ".env") if os.path.isfile(env_file): # read a local .env file env.read_env(env_file) password = env("DENIS_PASS") elif os.environ.get('GOOGLE_CLOUD_PROJECT', None): # pull .env file from Secret Manager project_id = os.environ.get('GOOGLE_CLOUD_PROJECT') client = secretmanager.SecretManagerServiceClient() settings_name = os.environ.get('SETTINGS_NAME', 'django_settings') name = f'projects/{project_id}/secrets/{settings_name}/versions/latest' payload = client.access_secret_version(name=name).payload.data.decode('UTF-8') env.read_env(io.StringIO(payload)) else: raise Exception('No local .env or GOOGLE_CLOUD_PROJECT detected. No secrets found.')
Enter fullscreen mode Exit fullscreen mode

There is two new imports: io and secretmanager

Great! It's finally time to deploy our app. To do so, run:

 $ gcloud app deploy Services to deploy: descriptor: [C:\Users\Nik\PycharmProjects\django-images-new\app.yaml] source: [C:\Users\Nik\PycharmProjects\django-images-new] target project: [indigo-griffin-376011] target service: [default] target version: [20230130t135926] target url: [https://indigo-griffin-376011.ey.r.appspot.com] Do you want to continue (Y/n)? y Beginning deployment of service [default]... #============================================================# #= Uploading 21 files to Google Cloud Storage =# #============================================================# File upload done. Updating service [default]...done. Setting traffic split for service [default]...done. Deployed service [default] to [https://indigo-griffin-376011.ey.r.appspot.com]
Enter fullscreen mode Exit fullscreen mode 
You can stream logs from the command line by running:
Enter fullscreen mode Exit fullscreen mode 
 $ gcloud app logs tail -s default
Enter fullscreen mode Exit fullscreen mode

Open your web app in your browser and test if it works:

 $ gcloud app browse
Enter fullscreen mode Exit fullscreen mode

If you get a 502 Bad Gateway error, you can navigate to Logs Explorer to see your logs.

If there's a 403 Permission 'secretmanager.versions.access' denied error, navigate to django_settings secret permissions and make sure the default App Engine service account has access to this secret. See solution on StackOverflow

And that is it! We have now our toy app fully up and running on Google app engine for world to see

Code can be found in this repo

p.s. whole this process is defined by great crew at testdrive.io High recommendation for all their writings!

Top comments (0)