Starting Out with Azure IoT Edge

In today’s article I’d like to discuss Azure IoT Edge and how you can get started with it. I will be explaining the benefits of Azure IoT Edge and providing a couple of demos to demonstrate how you can create a very simple IoT solution using an IoT Edge device.

Benefits of Azure IoT Edge

Azure IoT Edge is Microsoft’s fully managed service built on IoT Hub, which enables users to perform edge computing on one or more IoT Edge devices. Edge computing is the idea that the IoT devices (at the “edge” of the network) are the ones performing the heavy computing instead of server processes in the cloud.

Figure 1: Azure IoT Edge Diagram

This is beneficial as the individual devices can perform the calculation/processing/analysis and send the resulting insights to the cloud, instead of the devices sending all the raw data to the cloud for processing. Using edge computing results in reduced bandwidth costs, faster response times and reduced traffic.

Azure IoT Edge Components

There are three components that make up Azure IoT Edge:

Figure 2: Azure IoT Edge Components
  • IoT Edge Modules
  • IoT Edge Runtime
  • Cloud Interface

IoT Edge Modules are the Azure or third-party services that run in containers. You can also provide your own code. These modules are deployed to the IoT Edge devices and run locally on those devices. They provide the Edge “magic” as they can perform calculation/processing/analysis of your device data.

The IoT Edge Runtime is the code that runs on each IoT Edge device and manages the modules running on the device. The runtime also performs tasks such as maintaining security on the device, reporting module health to the cloud and managing communications between any downstream leaf devices and the IoT Edge device, between modules on the IoT Edge device and between the cloud and the IoT Edge device.

The cloud interface provides the ability to centrally manage the lifecycle for your IoT Edge devices. You can create and configure one or more workloads for your devices, as well as deploy them to all of your devices. You can also monitor your devices for any errors or devices that are not behaving properly.

Figure 3: IoT Edge Cloud Interface

Deploying an IoT Edge Module

The first demo I’d like to show you involves setting up a virtual IoT Edge device and deploying some sample code. We will also create an IoT Hub so the device can send its data to it.

Prerequisites

For both of these demos there are a couple of prerequisites that you will need. The first is an Azure subscription as we will be creating some cloud resources in these demos. You can sign up for a free account here.

The other prerequisite is installation of the Azure IoT Extension for Azure Cloud Shell. You can add this extension by launching a Cloud Shell session after logging into your Azure Portal. Once the shell window displays enter the following command to install the extension:

az extension add --name azure-iot

You only need to run this once.

Setup of IoT Edge Device

We will be using a virtual machine (VM) hosted in Azure as our IoT Edge device. I will run through the setup and configuration of this VM using Linux (there are similar steps if you wish to use a Windows VM).

There are two ways you can create a VM in Azure. The first way is to manually go through the steps in the Azure Portal to provision a VM. You should select the latest Ubuntu Server build for your image if you’re provisioning a Linux VM. This will give you a standard VM with no additional components installed. You would then be required to manually install the Azure IoT Edge components afterwards.

The second way to create this VM is to use the Azure Cloud Shell. I prefer this way as you can select a custom IoT Edge image that also automatically installs all the components needed to run IoT Edge on the VM (acting as a device). Accept the terms of use and create a VM by executing these commands:

az vm image terms accept --urn microsoft_iot_edge:iot_edge_vm_ubuntu:ubuntu_1604_edgeruntimeonly:latest
az vm create --resource-group IoTEdgeResources --name EdgeVM --image microsoft_iot_edge:iot_edge_vm_ubuntu:ubuntu_1604_edgeruntimeonly:latest --admin-username azureuser --generate-ssh-keys

Note that the commands expect you have a resource group created called IoTEdgeResources. The VM will be provisioned with a name of EdgeVM, and with an administrator username of azureuser and SSH public key authentication. You can see the command-line switch of –generate-ssh-keys which will generate the public and private keys and store them in the ~/.ssh folder in your Azure Cloud Shell instance.

Creating Azure IoT Hub and Device Registration

The next step is to provision an instance of Azure IoT Hub and register an IoT Edge device. Again, you can do this in two ways. The first is to manually run through the steps in the Azure Portal to provision IoT Hub and register your IoT Edge device.

The other way is to use the Azure Cloud Shell. Here are the commands to instantiate IoT Hub and register an IoT Edge device. Please note the resource group of IoTEdgeResources and make sure you enter your own name for the IoT Hub and Edge device. Take note of the –edge-enabled command-line switch, as this is what tells Azure to register a device as an IoT Edge device.

az iot hub create --resource-group IoTEdgeResources --name {hub_name} --sku F1 --partition-count 2
az iot hub device-identity create --hub-name {hub_name} --device-id {device_name} --edge-enabled

Once these commands finish, run the following command to display the device connection string and save it for a later step.

az iot hub device-identity show-connection-string --device-id {device_name} --hub-name {hub_name}
Figure 5: IoT Edge Device in Azure IoT Hub

Configure IoT Edge Device

Now that we have Azure IoT Hub and a registered IoT Edge device, we need to set the connection string with the value you saved previously from your registered IoT Edge device in IoT Hub. This will configure the VM to send its data to that registered device in Azure.

In your Azure Cloud Shell instance enter the following command, replacing for the IoT Edge device connection string:

az vm run-command invoke -g IoTEdgeResources -n EdgeVM --command-id RunShellScript --script "/etc/iotedge/configedge.sh '{device_connection_string}'"

Please note that running this command expects that the IoT Edge runtime is already installed on the VM. Since we used the Azure IoT Edge on Ubuntu image when provisioning the VM, this has already been installed. When complete, the output should look similar to this:

Figure 6: Set Connection String for IoT Edge Device

Once this command is complete you will need to connect to the VM using the public IP address that was given when the VM was created. This can be done right in the Azure Cloud Shell with this command:

ssh azureuser@{publicIpAddress}

The first time you do this you will be prompted to establish the authenticity of the host. Enter yes to continue connecting. This will also add an entry into a known_hosts file (so you are not prompted in the future).

Figure 7: SSH Host Authentication

Alternatively, you can use an SSH client (like PuTTY) to SSH into your VM. However, be aware that you need to copy the public and private SSH keys from your Azure Cloud Shell locally and create a .ppk file that PuTTY can use for authentication. Instructions on how to do this are located here.

Verify that the IoT Edge security daemon is running properly as a system service by executing the following command:

sudo systemctl status iotedge
Figure 8: IoT Edge Security Daemon

You can also view the running modules on the device. At this point you should only see the edgeAgent running.

sudo iotedge list
Figure 9: Running Modules on IoT Edge Device

Deploy IoT Edge Module

We can now deploy our module to the IoT Edge device. We are going to use a pre-built module from the IoT Edge section of the Azure Marketplace, which sends simulated temperature and pressure data to the cloud. You can also use your own custom code.

Note that we don’t have to access the device to deploy this module – we can perform the deployment right from the cloud. IoT Edge modules are executable packages implemented as containers. In order to deploy our module, navigate to your IoT Hub in the Azure Portal. Click on IoT Edge under Automatic Device Management from the left-hand pane.

Figure 10: Automatic Device Management

Click on your device in the list and select Set Modules from the upper bar. Click on the Add button to dropdown a menu selection. Click on +Marketplace Module to add a module from the marketplace.

Figure 11: Selecting a Marketplace Module

Enter simulated temperature sensor in the search box and select the results displayed. Click Routes to continue.

Figure 12: Simulated Temperature Sensor Module

You also need to configure the routes, which define how messages are passed between the module and IoT Hub. For our example, we want all the messages from the module to be sent to IoT Hub ($upstream), so add the following code for the Value for $upstream (if not auto-populated):

FROM /messages/* INTO $upstream

Click Review + Create and then Create to deploy the module.

After a few minutes if you refresh the page in the Azure Portal that lists the modules running on your device, you will see the SimulatedTemperatureSensor appear and show that its runtime status as running. Azure doesn’t push anything to your device when you add a module – the device checks Azure periodically for updates and when it sees that a module needs to be deployed it then pulls that module image from the cloud to start running it locally.

Figure 13: Simulated Temperature Sensor

Viewing the Data

You can view the simulated data from your IoT Edge device in two places. If you SSH into the VM and run the following command you can see the Simulated Temperature Sensor module executing and displaying the data it is sending to the cloud.

sudo iotedge logs SimulatedTemperatureSensor -f

You can also see the messages being received in your IoT Hub in the Azure Portal on the Overview page:

Figure 14: Azure IoT Hub Overview

Deploying to a Physical Device

The second demo I’d like to show you is how to deploy IoT Edge to a physical device. Using simulated devices is great for initial learning but I always like to actually deploy to a physical device since you will need to do this on an actual project! You will need a device that supports IoT Edge – I am using a Raspberry Pi 4 for this demo. You can use whatever device as long as the operating system can run containers. Please see this link for more details.

Most of the steps for the first demo still apply – the only difference is that instead of creating a virtual machine to simulate the IoT Edge device we will be using a real device (Raspberry Pi in our case). With that in mind, the first thing you should do is install the latest version of Raspian OS on your Pi.

Once you have the OS installed and you can SSH into the device, you will need to install the container runtime and the Azure IoT Edge components. The recommended container runtime is Moby. Run the following Bash commands on your device to update the package lists and install the Moby engine.

sudo apt-get update
sudo apt-get install moby-engine

You can optionally install the Moby command-line interface, which is helpful for development environments. When I ran this command the system indicated it was already installed when I installed the Moby engine.

sudo apt-get install moby-cli

After the container runtime is installed the next step is to install the Azure IoT Edge security daemon. This provides and maintains security standards on the device and bootstraps the device by starting the IoT Edge runtime when you boot up the device. Run the following Bash command to install the security daemon.

sudo apt-get update
sudo apt-get install iotedge

Once the security daemon is installed you then need to configure the IoT Edge device with the device connection string from your IoT Edge device in IoT Hub. This links your physical device to the device identity in the cloud. Open the configuration file at /etc/iotedge/config.yaml with this command:

sudo nano /etc/iotedge/config.yaml

Scroll down to the Manual Provisioning section and update the value of device_connection_string with your device connection string. Save the file when you are finished.

Figure 15: Configure Connection String for IoT Edge Device

You need to restart the security daemon to have the changes you made take effect, so run this command to restart IoT Edge on the device:

sudo systemctl restart iotedge

You can list the running modules on your device with this command. Please note that this point you should only see the edgeAgent module (since we haven’t deployed a module to the device yet).

sudo iotedge list

From this point you can follow the remaining steps from the first demo as they are identical. You would deploy your module from Azure IoT Hub in the portal, and once you do that you should be able to check the running modules on your physical device and see telemetry data (from the Simulated Temperature Sensor module) being sent to Azure IoT Hub.

Figure 16: IoT Edge Modules running on Physical Device

Congratulations! You have deployed an IoT Edge module to an actual physical device! You can now experiment further on creating other modules that can be deployed to your device.

I hope this article was helpful in explaining what Azure IoT Edge was and how you can deploy it to a simulated and real IoT Edge device. For more information on IoT Edge please visit https://docs.microsoft.com/en-us/azure/iot-edge/.

Thanks for reading!

Using Device Twins with Azure IoT Hub

In today’s article I’d like to discuss Device Twins and how you can use them with Azure IoT Hub. I will be discussing what Device Twins are, how they are used and then provide a code example of using them with Azure IoT Hub.

What are Device Twins?

Device Twins are JSON documents that store state information about IoT devices. This includes state information along with metadata about the device. Configurations and conditions on the device are also stored. Azure IoT Hub maintains a device twin for each registered device that is connected to that hub.

You can use Device Twins to store specific metadata about your devices in the cloud. For example, you can keep track of a device location by storing location metadata about that device in its Device Twin.

Device Twins can also be used to report state information about the device. This can be useful to determine if a device is connected to IoT Hub, or if it is connected to a WiFi network.

Additionally, the process of updating the firmware on one or more devices can benefit from Device Twins, as firmware installation progress can be reported to a back-end application.

Device Twin Structure

The JSON document that makes up a Device Twin has a few components that I will describe in this section. Here is an example of the overall JSON structure of a Device Twin:

Figure 1: Device Twin Structure

Tags

The Tags section is used exclusively by a back-end solution. Device applications cannot see or query tags at all, but a back-end solution does have the capability to read/write tag values to this section of the Device Twin. The information in this section can be whatever identifying data you need on the back-end solution side. Here is an example, which shows tags for the device location:

{
…
    "tags": {
        "deviceLocation": {
            "city": "Chicago",
            "building": "Willis Tower",
            "floor": "12",
            "room": "NE Conf 1"
        }
    },
…
}

Reported Properties

This section is used in coordination with the Desired Properties section to synchronize device configurations. The device application has the capability to read and write data to this section while a back-end solution can read information from this section as well as receive change notifications when data changes. Typically, this section is used by the device application to report its current state to the cloud.

For example, let us say that we want to report to a back-end solution the current WiFi information for the device. We would serialize the following JSON string into the proper format for Reported Properties and then report the state to the IoT Hub that the device is registered with.

{
…
    "properties": {
…
        "reported": {
            "wifi.wifiIP": “192.168.3.100”,
            "wifi.wifiMask": “255.255.255.0”
        }
    }
}
Figure 2: Sending Reported Properties (device app)

The IoT Hub that maintains the Device Twin for the registered device would update the information being received from the device. In our example, the WiFi IP and Mask values would be updated accordingly in the device’s device twin. A solution back-end can then query the Reported Properties to retrieve these values.

Figure 3: Reading Reported Properties (back-end solution)

Desired Properties

This section is used in coordination with the Reported Properties section to synchronize device configurations. A back-end solution has the capability to read and write data to this section, and the device application can read and receive change notifications for this section. Typically, this section is used by a back-end solution to communicate to the device of some change requested that the device application should handle.

For example, let us say that we want to change the interval time value on the device that is used for sending data to IoT Hub. A back-end solution could update the desired properties and that change would be communicated to the device. As shown in the JSON snippet below we can set the ‘sendDataInterval’ value in the Desired Properties section to ‘5000’ indicating we want to set the interval value on the device to 5 seconds.

{
…
    "properties": {
        "desired": {
            "sendDataInterval": "5000
        },
…
    }
}
Figure 4: Update Desired Properties (back-end solution)

When a change is made to the Desired Properties for a device, a notification event is sent to that device informing the device of the change. Code on the device would configure a Device Twin callback method that would be fired when the change notification is sent to the device.

Figure 5: Configure Device Twin Callback Method (device app)

The callback method would extract the JSON information which would contain the Desired Properties section of that device’s Device Twin record. The code would then be able to interpret what has changed and perform whatever necessary actions are required.

Figure 6: Device Twin Callback Method (device app)

A device application should perform whatever actions necessary to process the change in the Desired Properties and then send back an update (via the Reported Properties) to the cloud.

Device Identity Properties

This is the root of the JSON document and contains the read-only properties that specify the device identity. This information can be used to retrieve the device identity along with connection state and authentication type.

{
    "deviceId": "devA",
    "etag": "AAAAAAAAAAc=", 
    "status": "enabled",
    "statusReason": "provisioned",
    "statusUpdateTime": "0001-01-01T00:00:00",
    "connectionState": "connected",
    "lastActivityTime": "2015-02-30T16:24:48.789Z",
    "cloudToDeviceMessageCount": 0, 
    "authenticationType": "sas",
    "x509Thumbprint": {     
        "primaryThumbprint": null, 
        "secondaryThumbprint": null 
    }, 
    "version": 2, 
…
}

I hope this article was helpful in explaining the details of Device Twins. For more detailed information please visit https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-device-twins.

If you would like to download the full source code for this example, please visit my github repository here.

Thanks for reading!

Reading Sensors on the MXChip AZ3166 IoT Device

Recently I have been playing around with the MXChip IoT DevKit device (AZ3166, you can purchase one here) and I’ve noticed that while there are great code samples designed to get you started with the MXChip DevKit, there aren’t any basic examples showing you how to use the different sensors built into the device. This article aims to go through each of the available on-board sensors (temperature & humidity, pressure, accelerometer & gyroscope, and magnetometer) and give examples on how to use them.

I am assuming you already went through the steps to update to the latest firmware and connect to your WiFi network (steps are here; ignore the sections regarding connecting to Azure IoT Hub as we do not need that for this article). Also make sure your VSCode environment has the Azure IoT Tools extension installed.

There are 4 on-board sensors that come with the MXChip IoT DevKit device:

  • Temperature & Humidity
  • Pressure
  • Accelerometer & Gyroscope
  • Magnetometer

The MXChip IoT DevKit device is Arduino-compatible, and there is support for this hardware via the Microsoft Azure IoT Developer Kit in VSCode (via Arduino Board Manager). Once you’ve connected your MXChip device to your WiFi you can start creating Arduino sketches to upload the code for.

Setup

First off, you will want to create a new project in VSCode. Open VSCode and press F1 and type in Azure IoT Device Workbench in the textbox that appears.

Figure 1: Creating a New Project

Find the entry labeled Azure IoT Device Workbench: Create Project… and click on it. Enter a descriptive name for your project.

Figure 2: Project Name

Once you press Enter you will see another selection dropdown asking for the platform for this project. Choose Arduino as our project will be based on the Arduino platform.

Figure 3: Platform Selection

Pressing Enter will bring you to the next selection choice: the project template. This is where the different boards and service functionality are listed for the project being created. Since we are using the MXChip we want to select the choices that are available for the MXChip. You will notice that there are a couple of entries that have support for Azure IoT Hub or Azure Functions. For this article, we only need the base device support so select MXChip IoT DevKit selection.

Figure 4: Project Templates

At this point VSCode will create a new workspace with the project name you provided and create a device.ino file for your Arduino sketch.

If you look under the .vscode directory in your project (under Device) you will see 4 JSON files that were automatically created as part of this project. These files define what board the project is using along with other information such as the sketch name and the serial port being used.

Figure 5: Project Configuration

General Information

Since we are coding an Arduino sketch for our device, I thought I’d briefly go through some of the general code and setup that is needed before we delve into the sensor code specifics.

First off, there are the setup() and loop() methods, which need to be coded first. The setup() method is for code that is executed once when the sketch is run, while the loop() method runs on a continuous loop (after setup is complete).

There are a few initializations we need to perform when the code first runs. These are as follows:

  • Initialize the OLED display
  • Start the serial support
  • Initialize WiFi

First thing is to initialize the OLED display on the board so we can display text. Fortunately, the Microsoft Azure IoT Developer Kit library provides a Screen object that we can use (this is defined in the Arduino.cpp file):

Figure 6: Initializing Screen object

Also shown is the Screen.print() method which allows us to display text on the OLED display.

Next is to enable the serial port support for the board. This is done by another class in the Microsoft Azure IoT Developer Kit library called Serial. We want to utilize 115200 baud so we pass that number as the parameter to the Serial.begin() method to start our serial port.

Figure 7: Serial Support

We also want to make sure our WiFi connectivity is working properly. Even though our example in this article doesn’t explicitly use WiFi, it is a good practice to enable this and verify connectivity since we would need this for any communication with the cloud or over-the-air (OTA) updates, such as firmware updates.

Again, the Microsoft Azure IoT Developer Kit library files have support for WiFi for the AZ3166 chipset (see the AZ3166WiFi.cpp file) so all we have to do is utilize the WiFi class in our InitWiFi() method. This method enables WiFi for the board and checks if it connects successfully. If it does, it sets the hasWiFi flag to true and prints the IP address the device is using to the screen. The setup() method will return if the WiFi is not connected.

Figure 8: WiFi Initialization

The final initialization we need to perform is the individual sensors we are utilizing. We need to initialize the temperature & humidity sensor, the pressure sensor and finally the magnetometer sensor. In order to do this, we need to instantiate a helper class of type DevI2C which will handle the details of I2C peripherals. We also need to pass 2 parameters to the instantiation of this helper class which correspond to GPIO pins 18 and 17 respectively (D14=GPIO_18, D15=GPIO_17).

Figure 9: I2C Helper Class Instantiation

There is also the main loop() method which runs on a continuous basis (after setup() finishes executing). This method will repeat indefinitely and here is where we want the code to do our work. Please note that I have a line of code that calls a delay so the loop doesn’t continuously run over and over again without a pause. We use this delay to give some time for the device to perform any background tasks it hasn’t yet finished.

Figure 10: Main Program Loop

Temperature & Humidity Sensor

In order to read from the temperature and humidity sensor we need to initialize this sensor in the setup() method. This is done by instantiating a new class of type HTS221Sensor (which is part of the Microsoft Azure IoT Developer Kit library). This class takes a parameter of type DevI2C so we pass the pointer to the DevI2C variable we created earlier. We then call the init() method to initialize the sensor.

Figure 11: Initializing the Temperature & Humidity Sensor

Once we have initialized this sensor we can now use it in the main loop() method. There are a number of supported methods for this sensor (located here). We are reading the temperature and humidity sensor readings by calling getTemperature() and getHumidity(), respectively. These methods read the sensor and place the results into a float variable which we pass as a pointer to the methods. Please also note that we enable the sensor when using it and disable the sensor when finished.

Figure 12: Reading Temperature, Humidity Sensor

Pressure Sensor

In order to read from the pressure sensor we need to initialize this sensor in the setup() method. This is done by instantiating a new class of type LPS22HBSensor (which is part of the Microsoft Azure IoT Developer Kit library). This class takes a parameter of type DevI2C so we pass the pointer to the DevI2C variable we created earlier. We then call the init() method to initialize the sensor.

Figure 13: Initializing the Pressure Sensor

Once we have initialized this sensor we can now use it in the main loop() method. There are a number of supported methods for this sensor (located here). We are reading the pressure sensor reading by calling getPressure(). This method reads the sensor and places the results into a float variable which we pass as a pointer to the method.

Figure 14: Reading Pressure Sensor

Accelerometer & Gyroscope Sensor

In order to read from this sensor we need to initialize it in the setup() method. This is done by instantiating a new class of type LSM6DSLSensor (which is part of the Microsoft Azure IoT Developer Kit library). This class takes a parameter of type DevI2C so we pass the pointer to the DevI2C variable we created earlier. We then call the init() method to initialize the sensor. Please also note that we can enable the accelerometer, gyroscope, and pedometer individually if we want.

Figure 15: Initializing the Accelerometer & Gyroscope Sensor

Once we have initialized this sensor we can now use it in the main loop() method. There are a number of supported methods for this sensor (located here). Note that we read the positional axes (x, y, and z) for the accelerometer and gyroscope, while we read a step count for the pedometer.

Figure 16: Reading Accelerometer & Gyroscope Sensor

Magnetometer Sensor

In order to read from this sensor we need to initialize it in the setup() method. This is done by instantiating a new class of type LIS2MDLSensor (which is part of the Microsoft Azure IoT Developer Kit library). This class takes a parameter of type DevI2C so we pass the pointer to the DevI2C variable we created earlier. We then call the init() method to initialize the sensor.

Figure 17: Initializing the Magnetometer Sensor

Once we have initialized this sensor we can now use it in the main loop() method. There are a number of supported methods for this sensor (located here). Note that we read the positional axes (x, y, and z) for this sensor.

Figure 18: Reading Magnetometer Sensor

Code Execution

In order to run this code on our MXChip device, we first need to compile the code successfully. In VSCode, press F1 and enter Azure IoT Device Workbench: Compile Device Code and select the entry. This will compile the code for your Arduino sketch file.

Figure 19: Compiling Arduino Sketch File

Once the file compiles successfully, you are now ready to upload the code to the device. Again, press F1 and type Azure IoT Device Workbench: Upload Device Code and select the entry. You can alternatively click the Upload button in the upper right.

Figure 20: Uploading Code to Device

Once the code uploads successfully, it begins executing immediately. If you open the Serial Monitor in VSCode you can see the output, which should look something like this:

Figure 21: Code Execution Output

I hope this article was helpful in explaining the details of utilizing the different on-board sensors in the MXChip. If you would like to download the full source code for this example, please visit my github repository here.

Thanks for reading!

Using Azure Stream Analytics with IoT Devices

Azure Stream Analytics is Microsoft’s PaaS (platform-as-a-service) event-processing engine that allows you to analyze and process large volumes of streaming data from multiple incoming sources. You can configure different input sources including IoT devices, sensors or business applications for data ingestion. Delivery outputs can also be configured to send the processed data to those destinations for performing actions or further analytics.

Overview

Azure Stream Analytics works on the concept of jobs. These jobs consist of one or more inputs, a query and an output. Stream Analytics ingests incoming data from one or more of these configured inputs and then a query is applied to this data to filter, sort, aggregate or join other reference data over a period of time. This transformed data is then sent to the configured output for storage or further downstream processing.

In this article, we will be using an example where I have a physical IoT Device (an MXChip dev kit) sending temperature data for processing.

Resource Creation

You can create an instance of Stream Analytics in the Azure Portal. Login to your portal at https://portal.azure.com. Once authenticated, click on Create a Resource to start the creation process.

streamanalytics1
Figure 1: Azure Services

Enter Stream Analytics Job in the search box and press Enter or click the magnifying glass.

streamanalytics2
Figure 2: Creating Stream Analytics Job Resource

This will return a list of results that match your entry. Select Stream Analytics Job to create a new instance of Stream Analytics. A description page will appear describing what Stream Analytics is, along with available pricing plans and documentation resource links. Click Create to begin the creation process.

streamanalytics3
Figure 3: Stream Analytics Job Overview

You need to provide some details when creating a Stream Analytics job in Azure. First, you will need to specify a name for your Stream Analytics instance. This name only needs to be unique to your subscription.

There is also a dropdown for selecting the Azure subscription to create the Stream Analytics instance under. It is auto populated with your current subscription but if you have multiple subscriptions you can select from one in the list.

You also need to specify a resource group that the Stream Analytics instance is hosted under. You can either select from a list of existing resource groups or click Create New to create a new resource group immediately and use that for your Stream Analytics instance.

Location specifies where you will be creating the Stream Analytics instance. Choose the location nearest to your physical location for best performance.
Hosting environment specifies where the Stream Analytics job will be deployed. Cloud specifies that the job will be hosted in the cloud (aka – the Azure Portal) while Edge specifies an on-premise IoT Edge gateway device.

Streaming Units determine the number of computational resources the Stream Analytics job will use when processing a query. Since the jobs perform all processing in memory, increasing this value will increase the amount of CPU and memory resources to process queries. This can be useful for scaling up and providing low latency stream processing.

streamanalytics4
Figure 4: Create Stream Analytics Job Details

Once created, your Stream Analytics job is stopped by default. In order to configure your job for processing you need to create three (3) things:

  • Inputs
  • Queries
  • Outputs

Let’s review each of these in detail.

Inputs

The first thing to configure are inputs. Inputs define a connection to an existing data source, and Stream Analytics accepts incoming data from these inputs. You can choose from three different input types: Azure Event Hubs, Azure IoT Hub, or Azure Blob Storage.

You can navigate to the Inputs page by clicking Inputs under the Job Topology section (on the left). This page will display a list of configured inputs while also allowing you to create a new input.

streamanalytics5
Figure 5: Stream Analytics Inputs

There are also two types of inputs: Stream and Reference. Stream inputs define an unbounded sequence of data events over time. This is the stream of input data coming from an input source and Stream Analytics requires at least one of these input types to be defined.

Reference inputs define a static set of data or data that slowly changes. This input type is typically used for data lookups or data correlation and is optional depending on your needs. An example would be having a lookup table that you would join with data from your stream input. Currently Azure Blob Storage and Azure SQL Database are supported as input sources for reference data.

For our example, we will be connecting a stream input of type IoT Hub to an existing Azure IoT Hub instance. You are prompted for an input alias name for this input source, along with selection of the subscription and IoT Hub you are specifying. There are also entries to specify IoT Hub routing endpoint to use as well as event serialization format and encoding.

streamanalytics6
Figure 6: Adding IoT Hub Input

Clicking Save will create the input and run a connection test to verify that Stream Analytics can connect to the input properly.

Outputs

Outputs must also be configured in order for the Stream Analytics job to function properly. Stream Analytics needs to send the transformed/processed data to a destination. Configuring one or more output satisfies this requirement.
There are a number of different outputs you can choose from, and are listed here:

  • Event Hub
  • SQL Database
  • Blob Storage/Data Lake Storage Gen2
  • Table Storage
  • Service Bus Topic
  • Service Bus Queue
  • Cosmos DB
  • Power BI
  • Data Lake Storage Gen1
  • Azure Function

Additional information is available for each of these types here.

streamanalytics7
Figure 7: Stream Analytics Outputs

For our example I will be configuring an output to Cosmos DB. Adding an output of this type displays a blade that allows entry of the details of the output being created. You need to enter an alias name for the output, along with the choice of entering the Cosmos DB setting manually or selecting from your Azure subscription. We are selecting an existing Cosmos DB resource that was created previously. Please note that you must have previously created a container in Cosmos DB before specifying the Container Name and Document Id values.

streamanalytics8
Figure 8: Adding CosmosDB Output

Queries

Once you have one or more inputs and outputs configured, you can now create a query to process the incoming data. Queries are the mechanism used by Stream Analytics to transform the incoming data in real time. These queries are written using Stream Analytics Query Language which are similar to SQL statements.

Structurally the query has a SELECT statement which selects data from one or more configured inputs and uses an INTO clause to specify an output to emit the transformed data to. There are also FROM, JOIN and WHERE clauses which help specify and filter the incoming data and perform processing against it.

In our example, when you click on the Query section under Job Topology, a page is displayed that shows the current query for this Stream Analytics job. You are able to edit in the window to specify your select query statements. Notice that we are using SELECT * wildcard to specify all fields from the input source and saving those to the specified output.

streamanalytics9
Figure 9: Stream Analytics Queries

You can see preview data displayed from our input source (IoT Hub device):

streamanalytics9-1
Figure 10: Input Preview Data

This querying editor also has the ability to perform discrete field selection and filtering. So, for example if we wanted to specify certain fields we could modify our query to look like this:

streamanalytics10
Figure 11: Discrete Field Selection in Queries

Additionally, if we wanted to filter the incoming data we have that ability as well. For example, if we wanted to exclude any incoming temperature data that did not exceed 100 we could specify a filter like this:

streamanalytics11
Figure 12: Query Filtering

Once we have the inputs, outputs and queries configured we are ready to turn on our Stream Analytics job. Looking at the Overview page we can see at a glance the configured inputs, outputs and what the query is specified as.

When we start the Stream Analytics job and wait a few minutes, we should begin to see data coming in and being processed. The Monitoring and Resource Utilization graphs would show some data coming through as it is being processed.

streamanalytics12
Figure 13: Stream Analytics Graphs

We can also see our data appearing in our CosmosDB database, as shown here:

streamanalytics13
Figure 14: CosmosDB Data Records

Conclusion

In conclusion, Azure Stream Analytics provides the ability to process complex events in real time, with the purpose of providing real time analytics to incoming data sources. We also showed an example on how to create and configure Azure Stream Analytics to accept incoming IoT device telemetry data and process and save it into a CosmosDB database. I hope this article has been helpful and please click here for more information.

Cheers!

Azure IoT Hub Messaging

The Azure IoT Hub service provides the ability for bi-directional communication with your devices. You can use Azure IoT Hub to send messages from your devices to your IoT cloud solutions, as well as send commands from your IoT cloud solutions to your devices. This article will describe the different types of communications possible with Azure IoT Hub and how you can use them.

Prerequisites

Before you can send/receive messages between Azure IoT Hub and your devices you need the following:

  • An Azure account. Sign up for a free trial here
  • Provision an instance of IoT Hub
  • Create at least one device with your IoT Hub

Bi-directional communication means just that – in both directions. What that means for Azure and IoT is that you either send data from your devices to IoT Hub, or data or commands from IoT Hub to your devices. I will describe the different options to accomplish this, as seen below:

  • Device-to-Cloud messages
  • Cloud-to-Device messages
  • Device Twins
  • File Upload
  • Direct Methods

Device-to-Cloud Messages

Sending information from your IoT devices to the cloud is supported by Azure IoT Hub with device-to-cloud (D2C) messages. Your devices can send telemetry time series and alert data to IoT Hub to be routed to other services for processing.

Azure IoT Hub comes with a built-in endpoint that back-end services can use to read telemetry data being sent from your devices. This endpoint is compatible with Event Hubs and you can also use standard IoT Hub SDKs to read from this endpoint.

When D2C messages are sent from your devices to IoT Hub, by default they are routed to this built-in service endpoint (messages/events). This endpoint is exposed to your devices with MQTT, AMQP and HTTPS protocols. You also have the ability to route these messages to custom endpoints that you create.

Messages can be routed to multiple endpoints, so if you create custom endpoints and setup the routing queries you can have a message be routed to more than one endpoint. IoT Hub currently supports the following custom endpoints:

  • Azure Storage (Blob or Data Lake Gen2)
  • Service Bus Queues
  • Service Bus Topics
  • Event Hubs

Here is an example route that filters out messages that have a temperature alert flag specified and routes the messages to the default events endpoint.

Figure 1: IoT Hub Routing Example

Please note that the built-in Event Hubs endpoint (messages/events) is the fallback route, so if incoming device messages do not satisfy query conditions for any custom endpoints they will be sent to this fallback route. You must enable the fallback route capability when you turn on message routing or your incoming messages will not be routed properly.

D2C messages received are stored temporarily by IoT Hub up to 7 days. They are lost if not processed by a configured back-end service. Messages can also have a maximum size of 256KB for each message. The throughput of what an IoT Hub can handle depends on the pricing tier. Details are here.

Additional details on D2C communications can be found here.

Cloud-to-Device Messages

Sending one-way communications from a cloud solution to one or more devices is possible with cloud-to-device (C2D) messages. Your cloud solutions can send notifications to devices, or commands that can alter the state of devices (device twins).

Azure IoT Hub will send messages to devices through a service-facing endpoint (messages/devicebound), and the device(s) then receive the messages through a device-specific endpoint (devices/{deviceid}/messages/devicebound).

Azure IoT Hub provides a guarantee that messages will be sent to devices at-least-once. In order to do this, IoT Hub persists C2D messages in per-device queues. These messages persist in the queues until the devices explicitly acknowledge completion (aka – they received the message successfully). Once acknowledged, IoT Hub removes the message from the appropriate per-device queue. This approach guarantees resiliency in the case of connectivity and/or device failures.

Additional details on C2D communications can be found here.

Device Twins

JSON documents that contain device state information are what is known as device twins. These documents store the metadata, configurations and conditions for a device. Azure IoT Hub maintains a device twin for each device you connect to IoT Hub. There are multiple uses that device twins provide:

  • Store device-specific metadata in the cloud
  • Report current state information about the device
  • Synchronize the state of long-running workflows between the device and cloud solution
  • Querying device metadata, configuration or state

The structure of a device twin is shown in Figure 2:

IoTHubMessaging2
Figure 2: Device Twins

Tags are a section of the JSON document that the cloud solution can read/write to. Devices cannot see these tags or their values.

Desired Properties are a section of the JSON document used to synchronize device configurations or conditions. The cloud solution can set these properties and the device can read them. The device can also receive notifications that the desired properties have changed.

Reported Properties are a section of the JSON document also used to synchronize device configurations or conditions. Devices can set these properties and the cloud solution(s) can read them.

Root object properties define the device identity and also provide container objects for tags and the reported, desired properties sections.

Here is an example of a device twin:

{
    "deviceId": "testDevice1",
    "etag": "AAJWKSOAAAAAc=", 
    "status": "enabled",
    "statusReason": "provisioned",
    "statusUpdateTime": "0001-01-01T00:00:00",
    "connectionState": "connected",
    "lastActivityTime": "2020-02-28T12:44:48.976Z",
    "cloudToDeviceMessageCount": 0, 
    "authenticationType": "sas",
    "x509Thumbprint": {     
        "primaryThumbprint": null, 
        "secondaryThumbprint": null 
    }, 
    "version": 1, 
    "tags": {
        "$etag": "123",
        "deploymentLocation": {
            "building": "A",
            "floor": "1"
        }
    },
    "properties": {
        "desired": {
            "telemetryConfig": {
                "sendFrequency": "5m"
            },
            "$metadata" : {...},
            "$version": 1
        },
        "reported": {
            "telemetryConfig": {
                "sendFrequency": "5m",
                "status": "success"
            },
            "batteryLevel": 55,
            "$metadata" : {...},
            "$version": 4
        }
    }
}

The cloud solution backend can operate on the device twin using these operations (through HTTPS):

  • Retrieve Device Twin: Returns the JSON document for the device twin
  • Update Device Twin: Enables the partial update of the JSON document. Tags and Desired Properties can only be updated. Cloud solution can also add or remove properties in these sections. Setting a property to null removes it from the section. There is also the ability to completely overwrite the Tags and Existing Properties with new JSON.
  • Receive Device Twin Notifications: Allows the cloud solution to be notified when a change is made to the device twin. A route in the IoT cloud solution called twinChangeEvents must be created and configured as a route for this operation to succeed.

The device can operate on its device twin using these operations:

  • Retrieve Device Twin: Returns the JSON document for the device twin. Tags are not visible.
  • Update Device Twin: Enables the partial update of the JSON document. Reported Properties can only be updated. This operation works in the same way as the cloud solution updates a device twin.
  • Observe Desired Properties: The device can choose to be notified of updates to the Desired Properties when they occur.

There are size limitations on the different sections in the device twin, as seen here:

  • Tags: 8KB max size on values
  • Desired Properties: 32KB max size on each value
  • Reported Properties: 32KB max size on each value

As an example, if the cloud solution wants to change the sendFrequency value for the device it modifies the device twin value for sendFrequency, as seen here:

"desired": {
   "telemetryConfig": {
   "sendFrequency": "5m"
},
…

The device(s) would be notified of this change immediately or when they become connected (if offline). The device would then report back to the cloud solution the updated configuration on the device:

"reported": {
   "telemetryConfig": {
   "sendFrequency": "5m",
   "status": "success"
},
…

File Upload

IoT devices can start file uploads through a device-facing endpoint in IoT Hub. You can have your devices request file uploads to an associated Azure storage account in the cloud, with IoT Hub acting as a dispatcher for the uploaded file (requested by a device) to the Azure storage account. File uploads can be used for large files such as media files or large telemetry batches.

In order for file uploads to work, you must first associate an Azure storage account to IoT Hub. This can be done via the Azure Portal or programmatically through the IoT Hub REST APIs.

IoTHubMessaging3
Figure 3: File Upload Section

Figure 4: File Upload Configuration in IoT Hub

Once you have made this association you can write code in your device to upload files to the cloud. IoT Hub will handle the upload to its associated blob storage account and provide notifications that you can monitor. Here is a great tutorial that reviews the steps needed to perform a file upload from your device.

Direct Methods

Direct methods provide an alternative way of communication with devices that work independently of the other communication options. Direct methods allow us to call a method on the device as a local service, which means we can get immediate feedback from the results of that call.

Direct methods are implemented on the device and use a request-response pattern when communicating. You can invoke a direct method through a service-facing URI and are a synchronous call. The invocation will either succeed or fail (after the timeout period), returning a result with an optional payload. The benefits/drawbacks of direct methods are as follows:

Benefits

  • Define multiple named direct methods
  • Pass an optional payload
    Returns a status code and optional payload
  • High throughput

Drawbacks

  • Only supports AMQP or MQTT
  • Doesn’t work if device is offline

As an example, here is how you would write code on your device to receive calls to a direct method:

await device.SetMethodHandlerAsync(“showMessage”, ShowMessage, null);
…


private static Task ShowMessage(MethodRequest methodRequest, object userContext)
{
…
}

Calling the SetMethodHandlerAsync class registers a named method handler with the device, which in this example the method name is ShowMessage. Parameters are the name of the message handler method as well as the reference to that handler method and an optional context object.
The definition of the ShowMessage handler method is also defined, which takes a MethodRequest object parameter and optional context object and returns a Task object.

Now that our devices are able to handle direct messages from the cloud, the only other thing to do is write code in our IoT cloud solution to call the direct method on our device. We can do this by defining a method to invoke the direct method call on our device, as seen in this console application example:

static async Task Main(string[] args)
{
   …
   var serviceClient = ServiceClient.CreateFromConnectionString(svcConnString);
   await CallDirectMethod(serviceClient, deviceId);
   …
}


private static async Task CallDirectMethod(ServiceClient sClient, string deviceId)
{
   var method = new CloudToDeviceMethod(“showMessage”);
   method.SetPayloadJson(“’This is a test direct method’”);

   var response = await sClient.InvokeDeviceMethodAsync(deviceId, method);

   Console.WriteLine($”Response status: {response.Status}, payload:{response.GetPayloadAsJson()}”);
}

As you can see, the Main method creates an instance to our IoT hub using the ServiceClient class. We then create a private method named CallDirectMethod(), which takes the instance of our ServiceClient class along with the Device ID value. This method instantiates the direct method via the CloudToDevice() class, which we specify the name of the direct message when instantiating (showMessage in our example). We can then set the payload JSON and asynchronously invoke the direct method on the device. A response is returned to let us know if the method ran successfully or not.

Conclusion

In conclusion, we described the different types of communications possible with Azure IoT Hub and how you can use them. I hope this article has helped you understand the different types of Azure IoT Hub messaging.

Cheers!

Azure IoT Central

Azure IoT Central is Microsoft’s SaaS (software-as-a-service) offering that allows you to quickly provision an IoT solution. This service reduces the burden and cost of developing, managing and maintaining enterprise-level IoT solutions, allowing you to focus more on your IoT data and how it can transform your business. I will be going through the steps to create an Azure IoT Central application and how easy it is to create and maintain an IoT solution.

You can use IoT Central to create a custom, cloud-based IoT solution without having to worry about the details of the underlying infrastructure. Customization of your solution can all be done from the browser in the application UI, and includes capabilities to:

  • Define the layout of properties and settings on a device template
  • Configure custom dashboards
  • Configure custom analytics to explore device data

 

Prerequisites

The only prerequisites needed are a Microsoft account (personal, work or school) as the creation process will ask you to login to your account, and an Azure subscription.

 

Getting Started

First you need to navigate to the Azure IoT Central Build site. You will notice a left-hand navigation menu which consists of several sections:

IoTCentral1

Figure 1: Azure IoT Central Menu

 

Clicking on the Build menu selection will take you to a page that displays a number of options to build your IoT application. You can start with a number of pre-generated templates that are geared towards specific industries or start with a blank application template.

Once you select either an industry-specific or custom template, a page displays that allows you to specify information about your application.

IoTCentral2

Figure 2: Creating an IoT Central Solution

The Application Name is pre-populated for you, as well as the URL. Please note that the application name and URL must be globally unique. The Application Template dropdown is only visible on custom applications, as you’re already selecting a specific template if you choose one of the industry-specific templates.

You also need to choose the pricing plan – you can try the solution for free for 7 days or select one of the other pricing tiers. These other tiers are based on how often you want to send messages from your devices. The Billing Info allows you to select the Directory (Azure Active Directory) and Azure Subscription you wish to have this solution hosted from. Location is also required as the system will provision the IoT Central solution resources in the closest region.

 

Web Application

Once the system provisions your IoT Central application you are taken to the Dashboard page, which displays some starting links such as adding devices, quick start demos, tutorials and documentation.

IoTCentral3

Figure 3: Dashboard page

 

Devices

Adding devices is easy in IoT Central – just click on the Devices tab and you are taken to a page that displays all the devices currently in the application. Adding a device is done by clicking on the +New button. A dialog window is displayed allowing you to configure the details about the new device being added:

IoTCentral4

Figure 4: Adding a new device

Device ID is a unique identifier that is used by the device to connect, while Device Name is a friendly name for the device. You can also toggle the Simulated option which specifies whether your device is simulated or real.

 

Device Templates

You can also create one or more device templates, which can then be used to create new devices that have the template characteristics. Click on the +New button on the Device Templates tab (under App Settings) and you can select the template type:

IoTCentral5

Figure 5: Device Templates

You have the ability to create a template for an IoT device, Azure IoT Edge device, or one of the pre-configured device templates listed. The list of certified devices (with IoT Plug and Play) are shown here and more are being added all the time.

Once you publish your device template, you can add devices based on that template. On the Devices tab you will notice a selection with the name of your device template that you can click on. This will display a list of the current devices that are based on this device template. Clicking the +New button will allow you to create a device based on this template.

IoTCentral6

Figure 6: Creating a Device using Device Template

 

Device Groups

Device Groups are a logical grouping of devices in the system based on specific attributes. You can create one or more Device Groups to filter one or more devices in your system. This is useful if you have hundreds (or thousands) of devices and you need to look at specific devices quickly.

Navigate to the Device Groups tab and click on the +New button to create a new device group. You will see a page that allows you to enter information for your new device group such as the Name and Scope. Adding one or more filters in the Scope area will define the filters needed to only display the device(s) you want to see.

IoTCentral7

Figure 7: Device Groups

 

Rules

You can create telemetry-based rules in IoT Central. This is done under the Rules section in the application. Clicking on the Rules tab will display a list of the current rules in the system. Create a new rule by clicking on the +New button. This will display a page that allows you to create a new rule.

IoTCentral8

Figure 8: Rules

Give the rule a name and then specify a target device template, which will define the devices to which the rule applies. If you need to specify less than all the devices in the device template you select, create one or more filters to filter the list of devices further.

Conditions are created to specify when the rule is triggered. You can specify one or more conditions the rule must meet to be triggered.

There is also a section for Actions. These are the things that happen once the rule has been triggered. You notice you can either send an email or call a web hook when the rule is triggered. You can create one or more actions if you choose.

IoTCentral9

Figure 9: Email Action

 

IoTCentral10

Figure 10: Webhook Action

 

Analytics

Azure IoT Central also has the capability to analyze device data including time series information. You can specify the telemetry values to display along with splitting the data collections by attributes (such as Device ID). Select the Device Group and Telemetry value(s) to analyze and click the Analyze button when you are finished.

IoTCentral11

Figure 11: Analytics

 

Jobs

You can use Azure IoT Central to manage your connected devices using jobs. Jobs are useful for performing bulk updates to your device properties or for running commands. Clicking on the Jobs tab will display a list of the existing jobs in the system, job status and when they were last run. Creating a new job will allow you to specify details of the job such as Name, what device group the job targets as well as the Job Type and detailed information.

IoTCentral12

Figure 12: Creating a Job

Running a job is done by clicking on the Run button. This will execute the job and return results of the operation.

IoTCentral13

Figure 13: Running a Job

 

Administration

The IoT Central application also allows administration of the website, including Users in the system, Roles, Pricing and the Device Connections. You also have the ability to permanently delete the solution.

IoTCentral14

Figure 14: Administration

 

Underlying Infrastructure

Since IoT Central is a SaaS (software-as-a-solution) offering, you do not have access to the underlying architectural components in Azure. What this means is that Microsoft fully hosts and supports the IoT Central application but they also manage the resources themselves. You do not have the ability to configure/customize/remove any of the individual Azure resources used in your IoT Central application. If you need to customize the IoT Central application beyond what is provided to you through the SaaS offering, then you need to consider Azure IoT Solution Accelerators which do provide the ability to manage and modify the underlying Azure components.

 

In conclusion, Azure IoT Central is a nice offering from Microsoft if you want to get your IoT solutions up and running quickly! Device provisioning and customizations are part of this service, as well as Rules, Jobs, and analytics for the device data. I hope this brief tour of Azure IoT Central has been helpful to you.

Cheers!

Message Routing with Azure IoT Hub

IoT solutions can be complex, especially if you have multiple devices that you need to send to more than one destination in your cloud solution. Azure IoT Hub provides the capability to send your device data to multiple destinations through message routing. This article will describe the available options with message routing and how they are used.

IoT solutions can be complex, especially if you have multiple devices that you need to send to more than one destination in your cloud solution. Azure IoT Hub provides the capability to send your device data to multiple destinations through message routing. This article will describe the available options with message routing and how they are used.

Prerequisites

Before you can use message routing with your devices you will need the following:

  • An Azure account. Sign up for a free trial here
  • Provision an instance of IoT Hub
  • Create at least one device with your IoT Hub

Incoming Messages

The devices in your IoT solution will be sending telemetry data into Azure IoT Hub. Conceptually, you normally think of these messages all being sent to one place and being handled/processed by the hub. However, what if you want these messages sent to different cloud services that may be part of your solution? In the past you might have had custom code to handle these messages and route them appropriately. That made your solution overly complex when you were required to handle all the details about these incoming messages, such as handling transient faults, how to deal with lost messages, and maintaining high reliability.
Fortunately, Microsoft has made this easier for us by providing message routing as part of Azure IoT Hub. IoT Hub will now take care of all those complex details for routing messages so we can focus on our custom IoT solution code. IoT Hub provides scalability for incoming messages, handling transient faults, and how to handle lost messages. The details of message routing are abstracted from us, and we only need to know how to configure the routing to specify where we want our messages to go.
MessageRouting1

Figure 1: Message Routing

 

Endpoints

There are a number of places you can route your device messages to by configuring IoT Hub. These are the final destinations of your data, allowing backend processing services to execute actions against the data you route to it. The different types of endpoints are as follows:

  • Event Hub
  • Service Bus Queue
  • Service Bus Topic
  • Storage

Also note that paid SKU hubs can have up to 10 additional endpoints while free SKU hubs can only have 1 additional endpoint.

Event Hub

Creating an event hub allows you to route your device data to a specific event hub resource to allow further/dedicated processing. Note that IoT Hub comes with a built-in endpoint that is compatible with this type. Here is an example of adding a new event hub endpoint to IoT Hub:

MessageRouting2

Figure 2: Event Hub Endpoint

 

Service Bus Queue/Topic

You can create a custom endpoint to a Service Bus Queue/Topic, which would then create a record that can be processed by an Azure function for example. Here is an example of adding a new Service Bus Queue endpoint to IoT Hub:

MessageRouting3

Figure 3: Service Bus Endpoint

Creating a custom endpoint to a Service Bus Topic works in the same way you would create one for a Service Bus Queue, but please note that your Service Bus Namespace must be the Standard (or higher) pricing tier for topics to be available. Also be aware that you cannot create a custom endpoint to a Service Bus Queue/Topic if they have sessions or duplicate detection enabled.

Storage

You can also create a custom endpoint that uses Azure blob storage, allowing the saving of incoming messages to an existing container (as blobs only). Here is an example of adding a new Azure Storage endpoint to IoT Hub:

MessageRouting4

Figure 4: Storage Endpoint

Batch Frequency specifies (in seconds) the maximum amount of time before the system will batch and write records to Azure Storage. The Chunk Size Window specifies the maximum size of blobs (in MB). Apache AVRO is the default encoding, although you can specify JSON.

Routes

Routes are created/configured to direct the appropriate data source to an endpoint for processing. When you create an endpoint you must specify the Name, Endpoint (built-in or custom), Data Source, and filter query.

MessageRouting5

Figure 5: Adding a Route

The Data Source field specifies where the data is coming from and must be one of the following selections:

MessageRouting6

 

You also have the ability to filter the incoming message data, which can be done using queries in the Routes. IoT Hub defines a common format for all device-to-cloud messaging, and filtering is done by writing a query against this common format. Here is an example:

{ 
  "message": { 
    "systemProperties": { 
      "contentType": "application/json", 
      "contentEncoding": "UTF-8", 
      "iothub-message-source": "deviceMessages", 
      "iothub-enqueuedtime": "2017-05-08T18:55:31.8514657Z" 
    }, 
    "appProperties": { 
      "processingPath": "{cold | warm | hot}", 
      "verbose": "{true, false}", 
      "severity": 1-5, 
      "testDevice": "{true | false}" 
    }, 
    "body": "{\"Weather\":{\"Temperature\":50}}" 
  } 
}

Querying on message system properties should prefix a ‘$’ symbol on the query. Queries on application properties do not need this symbol prefixed. For example, querying on the system property ‘contentEncoding’ would look like this:

$contentEncoding = 'UTF-8'

Querying on an application property would look like this:

processingPath = 'hot'

Please note that device-to-cloud messaging is not case-sensitive, so using unique property names is recommended. Here is an example of a query that filters records that have the application property ‘temperatureAlert’ = true.

MessageRouting7

Figure 6: Message Routing Queries

A fallback route is present so that any messages that do not satisfy any specified query conditions will be sent to this route. The default fallback route is the ‘messages/events’ route that is used by the built-in endpoint (compatible with Event Hub). You can turn on message routing in IoT Hub, which will allow you to create custom routes. Once a custom route is created, data stops flowing to the built-in endpoint unless you specifically create a route to that endpoint. Please be aware that if you delete all custom routes and you disable the fallback route, data will not be sent to any destination.

 

In addition to telemetry data, routes can be created for other events such as Device Twin changes, Device Lifecycle events and Digital Twin changes (currently in public preview). If you create a route with the data source set to one of these other events, IoT Hub will send messages to the appropriate endpoint for that event (Device Twin, Digital Twin, Device Lifecycle).

 

Conclusion

In conclusion, using message routing with Azure IoT Hub increases the flexibility, reliability and scalability of your IoT solution. Routing also enables you to keep track of specific events in your solution, such as device twins and device lifecycle. I hope this article has helped you understand the different options for message routing with Azure IoT Hub.

Cheers!

Creating and Configuring Azure IoT Hub

Microsoft provides the Azure IoT Hub service to assist in implementing Internet of Things (IoT) solutions. This fully managed Azure service enables reliable and secure bidirectional communication between your IoT devices and a back-end solution hosted in Azure. I am going to review the steps to create an IoT hub in Azure, along with how to register a device and configure tier and scaling options.

Prerequisites

The only prerequisite you need to follow along is an Azure account. You can sign up for a free trial here.

Creating an IoT Hub Instance

Azure Portal

Probably the easiest way to create an instance of IoT Hub is to use the Azure portal. You do this by first logging into the Azure portal with your Azure credentials, and then clicking on Create a Resource in the upper left. This displays a new blade with a number of choices under Azure Marketplace. Click on Internet of Things and you should see the IoT Hub selection, as seen here:

CfgIoTHub2

Clicking on IoT Hub will display a new blade that allows you to enter in details of your new IoT Hub, such as the Subscription, Resource Group, Region and IoT Hub name. The Subscription is auto filled with the subscription you are currently logged into. You can select a Resource Group to place the new IoT Hub in or create a new one. Region specifies the location that your IoT Hub will be created in, and the IoT Hub Name is a globally unique name you give to your IoT Hub.

CfgIoTHub3

Clicking Next: Size and Scale takes you to the next screen which allows you to select a pricing tier for your IoT Hub, along with the number of IoT Hub units allocated. The pricing tiers include a Free Tier as well as 3 levels of either Basic or Standard Tiers (B1-3, S1-3 respectively). The choice you make here determine a number of options relating mainly to data throughput and scaling. Details are here.

At the time of this writing, the highest pricing tier selection is S3: Standard Tier as shown below:

CfgIoTHub4

Clicking Review + create will take you to a summary page listing out the selection choices you made. Clicking Create here will instruct the Azure Portal to create the IoT Hub with the choices you selected.

CfgIoTHub5

Azure CLI

Another way to create an instance of IoT Hub is by using the command-line interface (CLI). Azure hosts an interactive shell environment called Azure Cloud Shell that you can use to execute CLI commands. You can launch Azure Cloud Shell by either going to https://shell.azure.com in your browser, or by clicking on the Cloud Shell button in the upper right in the Azure Portal.

CfgIoTHub9

Once you are in your Cloud Shell environment, you can execute the following command to create an instance of IoT Hub:

az iot hub create --name {your iot hub name} --resource-group {your resource group name} --sku S1

The only required parameters are the name of your IoT Hub along with the resource group it will be associated with. Insert a globally unique name for your IoT Hub along with the name of an existing resource group. The optional –sku parameter designates what pricing tier to create the IoT Hub with. Pricing details are located here.

There are additional optional parameters when creating an IoT Hub and are listed here.

Configuring IoT Hub Tiers

There are two tiers for Azure IoT Hub: Basic and Standard. You should consider what your IoT solution is designed to do before you select a tier. The difference between Basic and Standard tiers are the number of features they support. A Free tier is also provided, which has all the functionality of the Standard tier but has limited messaging allowances.

If your solution collects data and you are analyzing it centrally, then the Basic tier is probably the best choice. If you want to use advanced configurations to control your IoT devices remotely, or you want to distribute some of your workloads to the devices themselves, then the Standard tier is probably the best choice.

Each tier has three levels, listed in numerical order (1, 2, 3). The difference between these three levels is how much data they allow to be send from devices per day:

  • B1, S1 tiers: 400,000 messages/day/unit
  • B2, S2 tiers: 6 million messages/day/unit
  • B3/S3 tiers: 300 million messages/day/unit

The following features are available in all IoT Hub tiers:

  • Device-to-cloud telemetry
  • Per-device identity
  • Message routing, enrichments and Event Grid integration
  • HTTP, AMQP, and MQTT protocols
  • Device Provisioning Service
  • Monitoring and diagnostics

The following features are only available in the Free/Standard IoT Hub tier:

  • Cloud-to-device messaging
  • Device twins, Module twins, and Device Management
  • Device streams (preview at the time of this blog)
  • Azure IoT Edge
  • IoT Plug and Play (preview at the time of this blog)

Registering Devices

Once you have your IoT Hub instance created, you can register one or more devices identities that you intend to use with your IoT Hub.

Azure Portal

After you log in to the Azure Portal, navigate to your IoT Hub and click on the IoT Devices selection (under Explorers section):

CfgIoTHub6

This will display a list of the current devices created for the IoT Hub. You can click on the + New link towards the top of the page and a new blade will be displayed, allowing you to enter the following information about your device:

  • Device ID – enter a name for the device. This is used for authentication and access control
  • Authentication Type – select how the device will authenticate to IoT Hub. Symmetric Key is the default and allows entry of the primary/secondary keys (see below). X.509 Self-Signed and X.509 CA Signed allow you to select security certificates for the device (certificate created prior)
  • Primary/Secondary Keys – allows you to set a unique value for accessing the device. Disabled if auto-generate keys is checked (valid for Symmetric Key authentication type only)
  • Auto-generate keys – this will instruct Azure to generate unique values automatically for the primary/secondary keys (valid for Symmetric Key authentication type only)
  • Connect this device to an IoT Hub – specifies if this device will interact with this IoT Hub
  • Parent Device – specify if this device has a parent device. Used for IoT Edge

CfgIoTHub7

Clicking Save will instruct the Azure Portal to create the device with your selections. Once completed, you can access the device details by clicking on its name in the list of devices:

CfgIoTHub8

Azure CLI

You can also register device identities with your IoT Hub using Azure CLI. Execute the following command in your Cloud Shell instance, replacing {iothub_name} with your IoT Hub name and {device_id} with the name you wish to use for your device:

az iot hub device-identity create -n {iothub_name} -d {device_id}

The console will emit a JSON string upon success which give information about the device you just registered, such as primary/secondary key values, its connection state and whether it has IoT Edge capabilities:

CfgIoTHub10

You can verify in the Azure portal that the device is registered by navigating to the IoT Devices section in your IoT Hub. Your newly-registered device should be displayed in the list:

CfgIoTHub11

Additional parameters can be found here.

I hope this article has helped you understand what Azure IoT Hub is and how to create an instance and register your IoT devices to it, so you can begin sending data to Azure.

Cheers!

Beer Kegerator with IoT Part 2 – How A Beer to the Face is a Good Thing

Normally, getting splashed in the face with beer is not a good thing. I am going to show you how using your face with beer can be a good thing. This blog is a continuation of my previous blog on using IoT with a Beer Kegerator (link).

Deploying the first version of the IoT app was well received, and there were a lot of great feedback suggestions on how to improve the experience. I wanted to showcase some of the newer Microsoft technologies, and ended up choosing facial recognition support for the app. This would have great benefits, as it would allow accurate tracking of who is pouring the beer and when they requested a pour. Coupled with the volume tracking of the selected beer, this would provide excellent data capabilities on who is drinking beer, when the beer is poured, along with the consumption rates of the different beers provided.

I also wanted to expand the app to include all four beer lines that we have available, so this required a bit of a redesign of the app. In order to do this, I needed to have the app enable/disable any of the four lines when someone requests to pour a specific beer.

Additional Hardware

First off, I purchased the additional three flow sensors so that I had the ability to track volume on all four beer lines. Modifying the code to account for this was simple, as it was just a matter of designating additional GPIO pins to receive the digital pulses.

More interesting is how I was going to handle the selection of beer when a pour is requested. I needed to verify that only the selected beer was able to be poured, while the other lines would not work. This meant providing the ability to disable each of the beer lines when necessary. I solved this problem with a food-grade quality solenoid valve, which allows opening/closing control of the valve via an electrical signal.

solenoid

This valve is normally closed, which means that when powered off the valve is closed, thereby stopping the flow of beer in the line it’s connected to. Providing power to the valve (12 volts) opens the valve as long as the power is applied to it.

The next question is how do I control the 12 volt signal to each of the solenoid valves connected to the four beer lines? This can be done with a simple relay switch, which allows a control signal to toggle the 12v on or off to the solenoid valve. I found a simple circuit board with four relays on it:

relayswitches

This will work perfectly for my situation, as I can use 5v control signals from the raspberry pi to turn on/off the different beer lines (activating a solenoid via the relay board) when a pour is requested.

Software Changes

With the hardware issues resolved, we now turn to the changes needed in the software. In order to support and control four beer lines, changes are needed in both the IoT app on the Raspberry Pi, as well as creating a web app to handle administration.

IoT Kegerator App

The main changes to the IoT app are centered around how to display the four different beer lines that a user can choose from, along with the process to select a beer and have the facial recognition routines recognize the person, so the selected beer line can be opened for the pour.

The previous version of this app only handled one line, so I needed to leverage the touchscreen I am using with the Raspberry Pi, in order that a person can touch an icon of the beer they want.

BeerKegerator2

This will start the facial recognition process so they can be recognized. Once the app recognizes them, it will open their selected choice for 30 seconds for them to pour their beer. After 30 seconds, the system will close the line and be ready for the next person.

BeerKegerator3

The following code snippet shows how the app processes a request from a person, when they touch one of the beer icons:

BeerLine1ButtonClick_CodeSnippet

The first thing this method does is display the associated web camera feed and then calls the facial recognition routine to take a snapshot of the webcam and process it for recognition. If it returns with a valid registered person, the routine will open the solenoid valve for that beer line ONLY, by setting a GPIO pin which will drive a specific relay, which in turn enables the 12v signal to the solenoid valve to open it. The method will then set the 30 seconds and start a timer to count down to 0, where it will then reset for the next time.

ActiveTimerTick_CodeSnippet

ActiveTimerTick2_CodeSnippet

This timer method runs every second, and records the digital pulses coming from the open beer line flow sensor, for the 30 seconds the beer line is open for. After it reaches the end of the 30 seconds, it will then take the total of the recorded digital pulses and send that info to Azure for saving into the database. The method will then reset some counter variables and turn itself off.

Facial Recognition

The final change needed for the IoT Kegerator App is adding support for facial recognition. Since I wanted to showcase Microsoft technology as part of this app, it made sense to leverage the Azure Cognitive Services suite of functionality that they provide (more info here). Specifically, I will be using the Face API that is part of the Computer Vision area.

The Face API is designed to detect and analyze faces/facial features in images. This will work for my implementation, as I can save the picture that is taken by the webcam when a person is requesting a beer. The trick then is to compare that image taken from the webcam against a “trained” model of images for a registered user. If recognized, the API reports that back to the app, which in turn will open the beer line for the person. Here is a code snippet showing how the Face API is called and leveraged to recognize a face when the app takes a picture.

FaceAPI_CodeSnippet

Please note that the CustomPerson class is based off the Face API Person class. The UploadandDetectFaces method calls the Face API to detect any faces in the webcam picture, and returns an array of detected faces. I then call the IdentifyAsync method against the array of detected faces to see if any correspond to any registered user face models that I trained previously. If the Face API determines there is a match I get additional details about the recognized face and return that to the calling method.

Here is a screenshot of the final redesigned app (with hardware):

BeerKegerator1

In conclusion, now that I’ve implemented this latest version of the IoT Kegerator app, I am able to track the consumption of each beer line, along with how much beer each registered user is consuming. I also have the ability to track time series data on when beer is being consumed, along with consumption rates of each beer on the four lines. Here is a quick example of a monthly consumption report:

ConsumptionReport

Here’s to getting a beer to the face!

Beginning IoT – Installing Windows 10 IoT Core on an x86/x64 Device

This is the second part in a 2-part series on how to install Microsoft Windows 10 IoT Core on an Internet-of-Things (IoT) device. Part 1 described how to install Windows 10 IoT Core on a Raspberry Pi 3 device. This article will focus on the steps required to install Windows 10 IoT Core on an x86/x64 device.

 

What is Windows 10 IoT Core?

Windows 10 IoT Core is a version of Microsoft’s Windows 10 operating system that has been optimized for smaller devices that can run on either ARM or x86/x64 devices. These devices can run with or without a display device.

When we talk about the different IoT devices, the processor type needs some explanation. ARM devices are called Advanced RISC Machines, with RISC standing for Reduced Instruction Set Computer. What this means is that the processor has been slimmed down to only include a reduced set of commands it can process. While this means that the processor can’t do certain things, it requires a low amount of power to execute what it can do, so that translates to increased battery life. The Raspberry Pi is classified as an ARM device.

Devices with the x86/x64 architecture are classified as CISC processors, which stands for Complex Instruction Set Computer. These processors do not have their instruction sets slimmed down, so they can perform more complex operations, at the cost of increased power consumption (and therefore lower battery life). Intel’s Baytrail devices running the Intel Atom processor E3800 is an example of an x64 device.

https://www.intel.com/content/www/us/en/embedded/products/bay-trail/overview.html

 

Prerequisites

Before you can install on an x86/x64 device, you need to make sure you have a PC that is running Windows 10 1507 (version 10.0.10240) or higher. You can find out what version you are running by clicking on the search box (next to the Start button) and typing ‘winver’. This will display a dialog as shown here:

1

 

For certain x86/x64 device you can use IoT Core Dashboard to run through the installation process. However, since the process is similar using IoT Core Dashboard (which I already covered in Part 1), I am going to go through the installation process steps using the Windows ADK IoT Core Add-Ons.

You will need an IoT device to install on – here are some options that Microsoft supports:

https://docs.microsoft.com/en-us/windows/iot-core/learn-about-hardware/socsandcustomboards

 

We will be using the Intel Atom E3800 (aka Baytrail) built on an industrial PC for examples in this article. Typically we would need a micro SD card for storage but since the industrial PC comes with onboard memory storage, we do not need an SD card.

 

For software, we will need to install the following on the PC we build the image on:

 

Installation Steps

Once you have the prerequisites installed, you are now ready to begin the installation process. We will be building a basic image, which involves combining the Windows IoT Core packages along with a board support package for the target hardware (Baytrail device) into a flashable file (FFU file).

First off, you need to set your OEM name, which will help you distinguish your created packages from packages other manufacturers have created. Edit the setOEM.cmd file, located at C:\IoT-ADK-AddonKit and set the OEM_NAME variable accordingly. Please note you can only use alphanumeric characters.

4

 

You should now open the IoTCoreShell.cmd file, which is a specialized command-line window that you will be doing a lot of the work for building the image. This file is located in the directory where you installed the IoT Core ADK Add-Ons (C:\IoT-ADK_AddonKit). Open an administrator-elevated privilege command window and navigate to the C:\ IoT-ADK_AddonKit directory. Type in IoTCoreShell.cmd to open the IoT Core Shell. This application will prompt you to select the architecture you are working with (1 for ARM, 2 for x86, 3 for x64). We are creating an x64 image so select x64.

5

 

At this point, you need to install certificates which will be used to sign the package binaries. Since this article is focused on a test image, you can run the installoemcerts.cmd command to install test certificates in the root certificate store of the PC you are building the image on. This only needs to be done the first time you are building an image.

6

 

The next step is to extract the board support package (BSP) files for the device you are building an image for, and run the buildpkg.cmd command to build the package files used in creating the image. You can download and extract this BSP zipfile and copy to C:\IoT-ADK-AddonKit\Source-x64\BSP directory to begin using for building an image.

7

 

You can now begin to create the packages and build the FFU image for the Baytrail x64 device. Go back to the IoTCoreShell.cmd command window and enter buildpkg all which will build the .cab files for all the BSP directories the program sees under C:\IoT-ADK-AddonKit\Source-x64\BSP. Please note that if you had selected x86 when you ran the IoTCoreShell.cmd, running this command to build all the packages would look in the C:\IoT-ADK-AddonKit\Source-x86\BSP to build any BSP files located there.

8

 

Once the program finishes building all the BSP packages, you can now create a new project by entering the following, where Product_Name and BSP_Name are the name of the product you would like and the BSP name, respectively.

newproduct <Product_Name> <BSP_Name>

So, for example, entering newproduct MyBayTrailDevice BYTx64 will create a project and its files under C:\IoT-ADK-AddonKit\Build\amd64\MyBayTrailDevice for the BYTx64 board support package files.

9

10

You are now ready to build the actual FFU flashable image file. This can be done by entering buildimage <Product_Name> Test, replacing Product_Name with your product name (MyBayTrailDevice in our example). The second parameter specifies whether you are building a Test or Retail image. This process takes about 20-30 minutes to complete and once finished you will have a file named flash.ffu created under the C:\IoT-ADK-AddonKit\Build\amd64\ MyBayTrailDevice\Test subdirectory.

11

 

If you encounter any errors, the buildimage process will error out and specify a log file that has detailed information on the error.

Now that you have a flashable FFU image file, you will need to flash it onto the IoT device you are working with. For our example, this is the Baytrail device and since it has onboard storage space we need to use a bootable USB thumbdrive with the FFU file on it. In order to create this, we will use Windows PE to create a bootable disk and then copy the flash.ffu file onto it. Here are instructions to create a bootable WinPE thumbdrive:

https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/winpe-create-usb-bootable-drive

 

Copy the flash.ffu file to the root of this bootable drive once you’ve created it. You are now ready to insert this USB thumbdrive to the IoT device and power it up. Make sure you specify in the IoT device’s BIOS to boot first from a USB drive.

12

WinPE will boot up and open a command window for you at the x: drive. Change to the d: drive and enter dir to see your flash.ffu file. WinPE comes with DISM, the Deployment Image Servicing and Management tool and we will be using this to flash the FFU file onto the IoT device. Enter the following in the command line to flash the FFU file:

dism.exe /apply-image /ImageFile:Flash.ffu /ApplyDrive:\\.\PhysicalDrive0 /skipplatformcheck

13-14

 

Once DISM has successfully completed the flashing process, you can power down the IoT device and remove the USB thumbdrive. Turn on the IoT device and have it boot normally off its storage. After a few minutes you should see the Windows 10 IoT Core startup screen and it should prompt you to select a language and whether you want Cortana activated. Once you make these selections the default application will appear.

15

 

Congratulations! You have successfully installed Windows 10 IoT Core on a x64 IoT device! Future steps can now be to modify the image to have it include your custom application or you can add drivers to the image if you need other functionality (such as Bluetooth or serial communications).

 

Enjoy!