Real-time Patients Health Data Processing with Azure Stream Analytics

Business Context and Aim:

In this POC, we are addressing a scenario where health monitoring devices are sending real-time patients’ health data to the cloud for analysis and storage. The aim is to showcase how Azure services can be utilized to collect, transform, and store this data securely and efficiently.

Let’s go through the detailed steps of implementing Azure Stream Analytics Proof of Concept (POC) for sending patients’ health data from an IoT device to an Azure SQL Database. 

Architecture of Real Time Streaming using Azure Stream Analytics:

Architecture of Real Time Streaming using Azure Stream Analytics - Addend Analytics

Step 1: Set up Azure Resources

1. Create an Azure IoT Hub:

The IoT Hub will serve as the central hub for receiving data from IoT devices and managing their communication.

  • Go to Azure Portal
  • Click on “+ Create a resource”, then search for “IoT Hub” and create a new IoT Hub.
  • Configure settings such as name, pricing tier, resource group, region, etc.
Create an Azure IoT Hub 1 - Addend Analytics
Create an Azure IoT Hub 2 - Addend Analytics
  • We have created an IOT Hub named “iothubstream1”.

2. Register an IoT Device:

Registering IoT devices allows secure and authorized communication with the IoT Hub.

  • Inside the IoT Hub, navigate to “IoT devices”.
  • Click on “+ New” to register a new IoT device.
  • Provide a device ID and optionally set authentication type.
Register an IoT Device 1 - Addend Analytics
  • We have registered an IoT device named “Patient-data-iot-P1”.
  • Note down the “Connection string – primary key” from the device’s settings.
Register an IoT Device 2 - Addend Analytics

3. Create an Azure SQL Database:

Storing patients’ health data in a structured manner for future analysis.

  • Create a new Azure SQL Database.
  • Configure settings like server, database name, pricing tier, etc.
Create an Azure SQL Database 1 - Addend Analytics
  • While Creating SQL database we can create a new Server as well as use our existing SQL server.
  • We will go with the basic tier so we can reduce costs.
Create an Azure SQL Database 2 - Addend Analytics

4. Set up Azure Stream Analytics:

Process and transform real-time data as it flows through the system.

  • Create a new Azure Stream Analytics job.
  • Configure settings such as job name, resource group, location, etc.
  • Choose an appropriate timestamp for event ordering.
  • Configure input, output, and query as described in the following steps.
Set up Azure Stream Analytics 1 - Addend Analytics
  • We have created a Stream Analytics job named “StreamAnalytics_01”.
Set up Azure Stream Analytics 2 - Addend Analytics

Step 2: Python Code for Generating Patients Health Data

1. Install Required Libraries

Develop a simulation of IoT device data generation.

  • Open a command prompt or terminal.
  • Run: pip install azure-iot-device.

2. Generate Patients Health Data

Simulate the health data generated by IoT devices.

  • Write a Python script to simulate health data generation.
  • Use libraries like random or faker to create realistic data.
  • Instantiate a DeviceClient using the device connection string.
  • Create a JSON payload containing health data (e.g., temperature, heart rate) and use the send_event method to send data.

3. Send Data to IoT Hub

Emulate IoT device communication with the IoT Hub.

  • Import necessary modules: from azure.iot.device import IoTHubDeviceClient, Message.
  • Create an instance of IoTHubDeviceClient using the device connection string.
  • Use the create_from_connection_string method to establish the connection.
  • Create a Message object containing your health data in JSON format.
  • Use the send_message method to send the data to the IoT Hub.
Send Data to IoT Hub 1 - Addend Analytics
Send Data to IoT Hub 2 - Addend Analytics
Send Data to IoT Hub 3 - Addend Analytics
  • Here is the Python code which will generate the Patients Health data and send this data to IoT Hub.

Step 3: Configure Azure Stream Analytics

1. Input Configuration:

Ingest data from the IoT Hub for further processing.

  • In your Stream Analytics job, click on “Inputs” and then “+ Add stream input”.
  • Choose “IoT Hub” as the source.
  • Provide the IoT Hub connection string and consumer group (if required).
Input Configuration - Addend Analytics

2. Output Configuration:

Store transformed data in a structured format for analysis.

  • Click on “Outputs” and then “+ Add” to create a new output.
  • Choose “SQL Database” as the sink.
  • Provide the SQL Database connection string and specify a table where the data will be stored.
  • To specify a table first you should create a table in the SQL database.
  • Map the output fields to the corresponding columns in the SQL table.
Output Configuration - Addend Analytics

3. SQL Query Configuration:

Process and transform incoming data as per business requirements.

  • Click on “Query” to write the SQL query that transforms incoming data.
  • Use the SELECT INTO statement to specify the fields to be selected and the output table.
  • You can apply filtering, aggregation, and formatting operations as needed.
SQL Query Configuration - Addend Analytics

Step 4: Start the Stream Analytics Job

1. Start the Job:

Begin the data processing pipeline.

  • Click on “Start” to begin the Stream Analytics job.
  • The job will start processing data from the IoT Hub according to your query logic.

2. Monitor Job Execution:

Ensure the data transformation is happening as expected.

  • Monitor the job’s progress in the Stream Analytics portal.
  • Check for any errors or issues in the job logs.
  • Adjust your query or configurations if needed.
Monitor Job Execution - Addend Analytics
Monitor Job Execution 2 - Addend Analytics

Step 5: Verification

Confirm that data is stored accurately for future analysis.

1. Verify Data in SQL Database:

  • Connect to your Azure SQL Database using a tool like Azure Data Studio.
  • Run a SELECT query on the table where the data is stored to ensure it’s being populated correctly.
Verify Data in SQL Database - Addend Analytics

Business Outcome:

Through this POC, we’ve demonstrated the ability to collect health data from IoT devices, process it in real-time using Azure Stream Analytics, and store it securely in an Azure SQL Database. This showcases the power of Azure services in healthcare applications, ensuring data accuracy and security while paving the way for more advanced health monitoring solutions.

Addend Analytics is a Microsoft Gold Partner based in Mumbai, India, and a branch office in the U.S.

Addend has successfully implemented 100+ Microsoft Power BI and Business Central projects for 100+ clients across sectors like Financial Services, Banking, Insurance, Retail, Sales, Manufacturing, Real estate, Logistics, and Healthcare in countries like the US, Europe, Switzerland, and Australia.

Get a free consultation now by emailing us or contacting us.