Azure Monitor | Data Collection Rule (Use case Scenario)

In my previous post I had discussed about Azure Monitor | Data Collection Rules.

https://hiyusuf.blogspot.com/2021/12/azure-monitor-data-collection-rules.html

In this article we will discuss with one of the use case scenario on implementing Data Collection rule.

Current Scenario:

One of the enterprise having multiple subscriptions across different regions, each subscriptions are linked with its own log Analytics Workspace, which are managed & controlled by IT support team of that region.

This Workspace consist of Logs / counters of VM's , VM scale sets.

Requirement: Now, this enterprise have a centralize team who wants to centrally monitor / needs to view / keep an eye on performance counters for each VM's in each subscriptions from all regions.

Proposed Scenario:

New proposed scenario will be as followed to achieve given requirements: 

- Create a new Log Analytics Workspace per Region (As workspace are region specific).

- Go to Azure Monitor | Data Connection Rule | Create New Rule (For each subscription level)

• Basic - Select the subscription to manage deployed resources and costs. Use resource groups like folders to organize and manage all of your resources.
• Resources - Pick a set of machines to collect data from. The Azure Monitor Agent will be automatically installed on these machines.(This will also enable System Assigned Managed Identity on these machines, in addition to existing User Assigned Identities (if any).)
• Collect & Deliver - Configure which data sources to collect, and where to send the data to.
• Data Source type - Select which data source type and the data to collect for your resource(s).
• Destination - Select the destination(s) for where the data will be delivered. ( Destination Type depends on what type of data source selected) example, if data source is selected as windows event logs -- destination type will only show Azure monitor logs & if data source is selected as performance counters -- destination type will show Azure monitor metrics & Azure monitor logs.
• Select the destination workspace (in one case the centralize workspace).
• Review & Create.

Azure Monitor | Data Collection Rules

 Azure Monitor | Data Collection Rules

Data collection rules in Azure Monitor

Data Collection Rules (DCR) define data coming into Azure Monitor and specify where that data should be sent or stored. This article provides an overview of data collection rules including their contents and structure and how you can create and work with them.

Data collection rules currently support the following input sources:

Azure Monitor Agent running on virtual machines, virtual machine scale sets and Azure Arc for servers.

 

Components of a data collection rule

A data collection rule includes the following components.

COMPONENTS OF A DATA COLLECTION RULE
Component	Description
Data sources	Unique source of monitoring data with its own format and method of exposing its data. Examples of a data source include Windows event log, performance counters, and syslog. Each data source matches a particular data source type as described below.
Streams	Unique handle that describes a set of data sources that will be transformed and schematized as one type. Each data source requires one or more streams, and one stream may be used by multiple data sources. All data sources in a stream share a common schema. Use multiple streams for example, when you want to send a particular data source to multiple tables in the same Log Analytics workspace.
Destinations	Set of destinations where the data should be sent. Examples include Log Analytics workspace and Azure Monitor Metrics.
Data flows	Definition of which streams should be sent to which destinations.

Data collection rules are stored regionally, and are available in all public regions where Log Analytics is supported. Government regions and clouds are not currently supported.

The following diagram shows the components of a data collection rule and their relationship:

Create DCR and associations with Azure PowerShell.

https://docs.microsoft.com/en-us/powershell/module/az.monitor/new-azdatacollectionrule?view=azps-7.0.0

Parameters have to be defined based on above PowerShell commands.

Azure VM Backup Architecture

 

Azure VM Backup Architecture

Let’s look at the Azure Virtual Machine Architecture.

Diagram: Azure VM backup Architecture, and the process in creating a backup as described in the content.

• Microsoft has an extension inside of every Azure VM

  • When you configure Backup, it communicates with the Azure Backup Service and associates itself to a policy.

  • It identifies itself to the Azure Backup Service as a VM, and that the service should back it up accordingly.

• When it’s time for backup per the backup policy, Microsoft sends a command to the Azure Backup extension and then Azure Backup orchestrates a VSS snapshot.

• Once the snapshot is available it goes to your local VM storage as an instant recovery snapshot which you can quickly recover from as it is in your VM storage.

• In the background, the snapshot is compared to a snapshot of a previous recovery point and moves only the incremental blocks via HTTPs into the recovery services vault.

• The recovery services vault has encryption enabled via server-side encryption (SSE), so the backup is encrypted at rest and is protected while in transit.

• When you secure your data via Azure Disk Encryption you are given keys, key encryption key (KEK) and BitLocker encryption key (BEK), which go to an Azure key vault, and are also backed up via Azure Backup.

  This is important because when you recover, you don’t have to worry about what keys you had when the backup was taken and the keys are restored for you to apply these keys on the recovered data.

Azure SQL Database & Managed Instance Service

 

Azure SQL Database Service

SQL Database is a fully managed Platform as a Service (PaaS) Database Engine that handles most of the database management functions such as upgrading, patching, backups, and monitoring without user involvement. Azure SQL Database is always running on the latest stable version of SQL Server Database Engine and patched OS with 99.99% availability.

With Azure SQL Database, you can create a highly available and high-performance data storage layer for the applications and solutions in Azure. SQL Database can be the right choice for a variety of modern cloud applications because it enables you to process both relational data and non-relational structures, such as graphs, JSON, spatial, and XML.

Deployment models

Azure SQL Database provides the following deployment options for an Azure SQL database:

Diagram of Deployment Options

• Single database represents a fully managed, isolated database. You might use this option if you have modern cloud applications and microservices that need a single reliable data source. A single database is similar to a contained database in Microsoft SQL Server Database Engine.

• Managed instance is a fully managed instance of the Microsoft SQL Server Database Engine. It contains a set of databases that can be used together. Use this option for easy migration of on-premises SQL Server databases to the Azure cloud, and for applications that need to use the database features that SQL Server Database Engine provides.

• Elastic pool is a collection of single databases with a shared set of resources, such as CPU or memory. Single databases can be moved into and out of an elastic pool.

Azure SQL Database Managed Instance

Managed instance is a new deployment option of Azure SQL Database, providing near 100% compatibility with the latest SQL Server on-premises (Enterprise Edition) Database Engine, providing a native virtual network (VNet) implementation that addresses common security concerns, and a business model favorable for on-premises SQL Server customers. The managed instance deployment model allows existing SQL Server customers to lift and shift their on-premises applications to the cloud with minimal application and database changes.

The following diagram outlines key features of managed instances:

Key Features

The managed instance deployment model is designed for customers looking to migrate a large number of apps from on-premises or IaaS, self-built, or ISV provided environment to fully managed PaaS cloud environment, with as low migration effort as possible. Using the fully automated Data Migration Service (DMS) in Azure, customers can lift and shift their on-premises SQL Server to a managed instance that offers compatibility with SQL Server on-premises and complete isolation of customer instances with native VNet support.

The managed instance deployment option aims delivers close to 100% surface area compatibility with the latest on-premises SQL Server version through a staged release plan.

Features and capabilities

Managed instance combines the best features that are available both in Azure SQL Database and SQL Server Database Engine.