Creating a utility to generate 100 MongoDB collections, each populated with 1 million random documents, and deploying it on Kubernetes involves several steps. This guide walks through the process, from setting up a Kubernetes environment to generating the collections and deploying the job in a dedicated namespace.
1. Setting Up Your Kubernetes Environment
Ensure you have a Kubernetes cluster (such as GKE, EKS, AKS, or Minikube) and configure kubectl to connect to it.
2. Create a Dedicated Namespace
To keep this deployment isolated, create a namespace called my-lab:
kubectl create namespace my-lab kubectl get ns my-lab 3. Deploy MongoDB on Kubernetes
Create a Persistent Volume (PV)
Create a mongo-pv.yaml file to define a persistent volume for MongoDB data:
apiVersion: v1 kind: PersistentVolume metadata: name: mongo-pv namespace: my-lab spec: capacity: storage: 10Gi accessModes: - ReadWriteOnce hostPath: path: /data/mongo Apply the PV:
kubectl apply -f mongo-pv.yaml Create a Persistent Volume Claim (PVC)
Define a persistent volume claim in mongo-pvc.yaml:
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mongo-pvc namespace: my-lab spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi Apply the PVC:
kubectl apply -f mongo-pvc.yaml Create a MongoDB Deployment
Define the MongoDB deployment and service in mongo-deployment.yaml:
apiVersion: apps/v1 kind: Deployment metadata: name: mongo namespace: my-lab spec: replicas: 1 selector: matchLabels: app: mongo template: metadata: labels: app: mongo spec: containers: - name: mongo image: mongo:latest ports: - containerPort: 27017 env: - name: MONGO_INITDB_ROOT_USERNAME value: "root" - name: MONGO_INITDB_ROOT_PASSWORD value: "password" volumeMounts: - name: mongo-storage mountPath: /data/db volumes: - name: mongo-storage persistentVolumeClaim: claimName: mongo-pvc --- apiVersion: v1 kind: Service metadata: name: mongo namespace: my-lab spec: type: ClusterIP ports: - port: 27017 targetPort: 27017 selector: app: mongo Apply the deployment:
kubectl apply -f mongo-deployment.yaml 4. Connect to MongoDB
Verify the MongoDB deployment by connecting to it:
kubectl exec -it <mongo-pod-name> -n my-lab -- mongosh -u root -p password 5. Verify Persistence
Scale down and then back up the MongoDB deployment to ensure data persists:
kubectl scale deployment mongo --replicas=0 -n my-lab kubectl scale deployment mongo --replicas=1 -n my-lab 6. Create a Python Utility for Collection Generation
Using Python, define a script to create collections and populate them with random documents:
import random import string import pymongo from pymongo import MongoClient def random_string(length=10): return ''.join(random.choices(string.ascii_letters + string.digits, k=length)) def create_collections_and_populate(db_name='mydatabase', collections_count=100, documents_per_collection=1_000_000): client = MongoClient('mongodb://root:password@mongo:27017/') db = client[db_name] for i in range(collections_count): collection_name = f'collection_{i+1}' collection = db[collection_name] print(f'Creating collection: {collection_name}') bulk_data = [{'name': random_string(), 'value': random.randint(1, 100)} for _ in range(documents_per_collection)] collection.insert_many(bulk_data) print(f'Inserted {documents_per_collection} documents into {collection_name}') if __name__ == "__main__": create_collections_and_populate() 7. Dockerize the Python Utility
Create a Dockerfile to containerize the Python script:
FROM python:3.9-slim WORKDIR /app COPY mongo_populator.py . RUN pip install pymongo CMD ["python", "mongo_populator.py"] Build and push the image to a container registry:
docker build -t <your-docker-repo>/mongo-populator:latest . docker push <your-docker-repo>/mongo-populator:latest 8. Create a Kubernetes Job
Define a job in mongo-populator-job.yaml to run the collection generation script:
apiVersion: batch/v1 kind: Job metadata: name: mongo-populator namespace: my-lab spec: template: spec: containers: - name: mongo-populator image: <your-docker-repo>/mongo-populator:latest env: - name: MONGO_URI value: "mongodb://root:password@mongo:27017/" restartPolicy: Never backoffLimit: 4 Apply the job:
kubectl apply -f mongo-populator-job.yaml 9. Verify Collection Generation
After the job completes, connect to MongoDB to examine the data:
kubectl exec -it <mongo-pod-name> -n my-lab -- mongosh -u root -p password In MongoDB:
use mydatabase show collections db.collection_9.find().limit(5).pretty() db.getCollectionNames().forEach(function(collection) { var count = db[collection].countDocuments(); print(collection + ": " + count + " documents"); }); Each collection should contain 1 million documents, confirming that the data generation job was successful.
Top comments (0)