Storage Device & Usage Monitor in Cloud Computing.pdf

Device & Usage Monitor Dr Hitesh Mohapatra Associate Professor School of Computer Engineering KIIT University

Contents • Storage Device: • Types of storage devices in cloud (block storage, object storage) • Cloud-based storage solutions (e.g., Amazon S3, Google Cloud Storage) • Storage management techniques • Data backup and redundancy • Usage Monitor: • Role of usage monitoring in cloud • Tools for cloud usage monitoring (AWS CloudWatch, Azure Monitor) • Metrics tracked (CPU, memory, storage, network) • Real-time vs historical usage monitoring • Benefits for resource optimization and cost control

Storage Device: Definition • Cloud storage is defined as a data deposit model in which digital information such as documents, photos, videos and other forms of media are stored on virtual or cloud servers hosted by third parties. • It allows you to transfer data on an offsite storage system and access them whenever needed. This article delves into the basics of cloud storage.

What is Cloud Storage? • Cloud storage is a cloud computing model that allows users to save important data or media files on remote, third-party servers. • Users can access these servers at any time over the internet. Also known as utility storage, cloud storage is maintained and operated by a cloud-based service provider. • From greater accessibility to data backup, cloud storage offers a host of benefits. The most notable being large storage capacity and minimal costs. • Cloud storage delivers on-demand and eliminates the need to purchase and manage your own data storage infrastructure. With “anytime, anywhere” data access, this gives you agility, global scale and durability.

Cont. • Cloud storage works as a virtual data center. It offers end users and applications virtual storage infrastructure that can be scaled to the application’s requirements. • It generally operates via a web-based API implemented remotely through its interaction with in-house cloud storage infrastructure. • To ensure the constant availability of data, cloud storage systems involve large numbers of data servers. • Therefore, if a server requires maintenance or fails, the user can be assured that the data has been moved elsewhere to ensure availability.

Types of Cloud Storage • Cloud services have made it possible for anyone to store digital data and access it from anywhere. • This means that cloud storage is essentially a virtual hard drive. From saving important data such as word documents, and video files, to accessing the cloud to process complex data and run applications – cloud storage is a versatile system. • Private cloud storage • Public cloud storage • Hybrid cloud storage • Community cloud storage

Storage Devices There are 3 types of storage systems in the Cloud as follows. • Block-Based Storage System • File-Based Storage System • Object-Based Storage System

Block-Based Storage System • Block-based storage in cloud computing is a type of data storage where data is stored in fixed- sized blocks (usually 512 bytes or multiples thereof). • Each block is treated as an individual storage unit, and unlike file storage, block storage doesn’t manage files or directories. • It provides raw storage volumes that can be attached to cloud-based virtual machines (VMs) or servers, allowing them to use it like a traditional hard drive.

Key Features/ Common Use Cases / Provider • High Performance • Flexibility • Persistent Data • Scalability • Resilience and Backup • Databases • Virtual Machines • Applications Requiring High Throughput • Amazon Elastic Block Store (EBS) • Google Cloud Persistent Disks • Microsoft Azure Disk Storage

File-Based Storage System • A file-based storage system (also known as file-level storage or file storage) is a type of data storage where data is stored and organized as files within directories and subdirectories, much like how files are stored on a personal computer. • Each file is treated as a complete entity and accessed through a hierarchical file system (e.g., NTFS, HFS+, or ext4). • File storage is typically used for unstructured data such as documents, media files, and backups.

File-sharing methods • Peer-to-Peer (P2P) model – A peer-to-peer (P2P) file sharing model uses peer-to-peer network. P2P enables client machines to directly share files with each other over a network. • File Transfer Protocol (FTP) – FTP is a client-server protocol that enables data transfer over a network. An FTP server and an FTP client communicate with each other using TCP as the transport protocol. • Distributed File System (DFS) – A distributed file system (DFS) is a file system that is distributed across several hosts. A DFS can provide hosts with direct access to the entire file system, while ensuring efficient management and data security. Hadoop Distributed File System (HDFS) is an example of distributed file system.

Network Attached Storage (NAS) • The standard client-server file-sharing protocols, such as NFS and CIFS, enable the owner of a file to set the required type of access, such as read-only or read-write, for a particular user or group of users. Using this protocol, the clients can mount remote file systems that are available on dedicated file servers. So, for example if somebody shares a folder with you over the network, once you are connected to the network, the shared folder is ready to use. There is no need to format before accessing it unlike in block storage. Shared file storage is often referred to as network-attached storage (NAS) and uses protocols such as NFS and SMB/CIFS to share storage.

Key Features/ Common Use Cases / Provider • Hierarchical Structure • Accessibility via Network Protocols • Simplicity • Locking Mechanisms • Less Granularity • File Sharing and Collaboration • Unstructured Data • Archiving • Amazon Elastic File System (EFS) • Google Cloud Filestore • Azure Files

Object based Storage systems for Cloud Services • Object-based storage systems are a type of data storage architecture designed to handle vast amounts of unstructured data, such as images, videos, backups, and large datasets. • Unlike block and file storage, object storage doesn’t store data in blocks or files, but rather as objects. • Each object contains the data itself, along with metadata and a unique identifier, which allows for easy retrieval, management, and scalability. • It is the most scalable form of storage, often used in cloud environments to store massive quantities of data.

Key Features/ Common Use Cases / Provider • Data Stored as Objects • Flat Storage Structure • Scalability • Access via HTTP/HTTPS • Durability and Redundancy • Cost-Effective • Backup and Archival • Static Web Content • Big Data and Analytics • Media Storage • Amazon S3 (Simple Storage Service) • Google Cloud Storage • Microsoft Azure Blob Storage

Storage Management • The Key Storage Management Operations consists of Storage Monitoring, Storage Alerting, and Storage Reporting. • Storage Monitoring provides the performance and availability status of various infrastructure components and services. • It also helps to trigger alerts when thresholds are reached, security policies are violated, and service performance deviates from SLA.

Storage Monitoring • Monitoring forms the basis for performing management operations. • Monitoring provides the performance and availability status of various infrastructure components and services. • It also helps to measure the utilization and consumption of various storage infrastructure resources by the services. • This measurement facilitates the metering of services, capacity planning, forecasting, and optimal use of these resources.

Cont. A storage infrastructure is primarily monitored for : • Configuration Monitoring • Availability Monitoring • Capacity Monitoring • Performance Monitoring • Security Monitoring

Storage Alerting • An alert is a system-to-user notification that provides information about events or impending threats or issues. Alerting of events is an integral part of monitoring. • Alerting keeps administrators informed about the status of various components and processes. For example, conditions such as failure of power, storage drives, memory, switches, or availability zone, which can impact the availability of services and require immediate administrative attention.

Storage Reporting • Like alerting, reporting is also associated with monitoring. Reporting on a storage infrastructure involves keeping track and gathering information from various components and processes that are monitored. • The gathered information is compiled to generate reports for trend analysis, capacity planning, chargeback, performance, and security breaches.

Data backup and redundancy • One of the main feature the storage systems became intelligent is by using the technique called RAID. • A group of disk drives which combinedly referred as a disk array are very expensive, have single point of failure and have limited IOPS. • Most large data centers experience multiple disk drive failures each day due to increase in capacity and decrease in performance. • To overcome these limitations, 25 years ago a technique called RAID is introduced for the smooth uninterrupted running of the data centers. • A properly configured RAID will protect the data from failed disk drives and improve I/O performance by parallelizing I/O across multiple drives. Refer link for RAID PPT: https://www.slideshare.net/slideshow/raid-255852376/255852376

Usage Monitor • A cloud usage monitor is a tool that tracks and analyzes the usage of cloud resources to optimize their utilization and minimize costs. • It provides visibility into resource usage patterns such as CPU usage, memory usage, network traffic, and storage usage. • It enables businesses to identify underutilized resources and make the necessary adjustments to optimize performance and reduce expenses.

Examples • Some examples of cloud usage monitoring tools include CloudWatch on Amazon Web Services, Azure Monitor on Microsoft Azure, and Stackdriver on Google Cloud Platform. • These tools allow businesses to track resource usage and performance metrics, set alarms and notifications for specific thresholds, and visualize data in customizable dashboards.

The cloud usage monitor process

Benefits • A cloud usage monitor provides businesses with valuable insights into their cloud resources, allowing them to make data-driven decisions that improve efficiency and reduce expenses. • By optimizing resource usage, businesses can improve performance and ensure they get the most value from their cloud investments. • Additionally, cloud usage monitoring can help businesses avoid unexpected costs and performance issues by providing early warnings of potential problems.

Types of Cloud Monitoring (Metrics) The cloud has numerous moving components, and for top performance, it’s critical to safeguard that everything comes together seamlessly. This need has led to a variety of monitoring techniques to fit the type of outcome that a user wants. The main types of cloud monitoring are: • Database monitoring • Website monitoring • Virtual network monitoring • Cloud storage monitoring • Virtual machine monitoring

Real-time vs Historical Usage Monitoring Real-time Monitoring: Provides up-to-the-moment data on the performance of cloud resources. It is vital for immediate alerts and quick responses to problems as they arise. • Example: If CPU usage spikes suddenly, real-time monitoring would trigger an alert and potentially auto-scale the resources. Historical Monitoring: Focuses on data collected over a longer period, useful for analyzing trends, forecasting resource needs, and conducting audits. • Example: Analyzing monthly data to determine usage patterns and optimize resources accordingly.

Benefits for Resource Optimization and Cost Control • Cost Control: By monitoring the usage of resources, organizations can identify underutilized resources and scale down, thus reducing costs. Over- provisioned resources lead to unnecessary expenses, which can be optimized through monitoring. • Resource Optimization: Ensures that the right number of resources are allocated to match demand. Monitoring helps detect resource wastage or the need for additional resources, ensuring optimal application performance. • Preventing Downtime: Continuous monitoring prevents downtime by detecting issues before they escalate, which also helps maintain service availability. • Auto-scaling: Monitoring tools enable the automatic scaling of resources based on predefined conditions, ensuring resources are allocated efficiently.

Questions Storage Device: 1. What are the different types of storage devices available in the cloud? 2. Can you explain the differences between block storage and object storage in cloud environments? 3. How does Amazon S3 differ from Google Cloud Storage in terms of features and use cases? 4. What are some common storage management techniques used in cloud computing? 5. Why is data backup and redundancy important in cloud-based storage solutions? 6. How does cloud storage ensure high availability and disaster recovery for data? Usage Monitor: 1. What is the role of usage monitoring in cloud computing? 2. Which cloud tools are commonly used for monitoring usage in cloud environments? 3. How does AWS CloudWatch differ from Azure Monitor in terms of capabilities? 4. What are the key metrics tracked in cloud usage monitoring (e.g., CPU, memory, storage, network)? 5. How does real-time usage monitoring differ from historical usage monitoring? 6. What are the benefits of usage monitoring for resource optimization and cost control in cloud environments? 7. In what ways can usage monitoring help prevent resource over-allocation and under-utilization?

Storage Device & Usage Monitor in Cloud Computing.pdf

In this document

More Related Content

What's hot

Similar to Storage Device & Usage Monitor in Cloud Computing.pdf

More from Hitesh Mohapatra

Recently uploaded

Storage Device & Usage Monitor in Cloud Computing.pdf