Using Azure Stream Analytics with IoT Devices

Azure Stream Analytics is Microsoft’s PaaS (platform-as-a-service) event-processing engine that allows you to analyze and process large volumes of streaming data from multiple incoming sources. You can configure different input sources including IoT devices, sensors or business applications for data ingestion. Delivery outputs can also be configured to send the processed data to those destinations for performing actions or further analytics.

Overview

Azure Stream Analytics works on the concept of jobs. These jobs consist of one or more inputs, a query and an output. Stream Analytics ingests incoming data from one or more of these configured inputs and then a query is applied to this data to filter, sort, aggregate or join other reference data over a period of time. This transformed data is then sent to the configured output for storage or further downstream processing.

In this article, we will be using an example where I have a physical IoT Device (an MXChip dev kit) sending temperature data for processing.

Resource Creation

You can create an instance of Stream Analytics in the Azure Portal. Login to your portal at https://portal.azure.com. Once authenticated, click on Create a Resource to start the creation process.

streamanalytics1
Figure 1: Azure Services

Enter Stream Analytics Job in the search box and press Enter or click the magnifying glass.

streamanalytics2
Figure 2: Creating Stream Analytics Job Resource

This will return a list of results that match your entry. Select Stream Analytics Job to create a new instance of Stream Analytics. A description page will appear describing what Stream Analytics is, along with available pricing plans and documentation resource links. Click Create to begin the creation process.

streamanalytics3
Figure 3: Stream Analytics Job Overview

You need to provide some details when creating a Stream Analytics job in Azure. First, you will need to specify a name for your Stream Analytics instance. This name only needs to be unique to your subscription.

There is also a dropdown for selecting the Azure subscription to create the Stream Analytics instance under. It is auto populated with your current subscription but if you have multiple subscriptions you can select from one in the list.

You also need to specify a resource group that the Stream Analytics instance is hosted under. You can either select from a list of existing resource groups or click Create New to create a new resource group immediately and use that for your Stream Analytics instance.

Location specifies where you will be creating the Stream Analytics instance. Choose the location nearest to your physical location for best performance.
Hosting environment specifies where the Stream Analytics job will be deployed. Cloud specifies that the job will be hosted in the cloud (aka – the Azure Portal) while Edge specifies an on-premise IoT Edge gateway device.

Streaming Units determine the number of computational resources the Stream Analytics job will use when processing a query. Since the jobs perform all processing in memory, increasing this value will increase the amount of CPU and memory resources to process queries. This can be useful for scaling up and providing low latency stream processing.

streamanalytics4
Figure 4: Create Stream Analytics Job Details

Once created, your Stream Analytics job is stopped by default. In order to configure your job for processing you need to create three (3) things:

  • Inputs
  • Queries
  • Outputs

Let’s review each of these in detail.

Inputs

The first thing to configure are inputs. Inputs define a connection to an existing data source, and Stream Analytics accepts incoming data from these inputs. You can choose from three different input types: Azure Event Hubs, Azure IoT Hub, or Azure Blob Storage.

You can navigate to the Inputs page by clicking Inputs under the Job Topology section (on the left). This page will display a list of configured inputs while also allowing you to create a new input.

streamanalytics5
Figure 5: Stream Analytics Inputs

There are also two types of inputs: Stream and Reference. Stream inputs define an unbounded sequence of data events over time. This is the stream of input data coming from an input source and Stream Analytics requires at least one of these input types to be defined.

Reference inputs define a static set of data or data that slowly changes. This input type is typically used for data lookups or data correlation and is optional depending on your needs. An example would be having a lookup table that you would join with data from your stream input. Currently Azure Blob Storage and Azure SQL Database are supported as input sources for reference data.

For our example, we will be connecting a stream input of type IoT Hub to an existing Azure IoT Hub instance. You are prompted for an input alias name for this input source, along with selection of the subscription and IoT Hub you are specifying. There are also entries to specify IoT Hub routing endpoint to use as well as event serialization format and encoding.

streamanalytics6
Figure 6: Adding IoT Hub Input

Clicking Save will create the input and run a connection test to verify that Stream Analytics can connect to the input properly.

Outputs

Outputs must also be configured in order for the Stream Analytics job to function properly. Stream Analytics needs to send the transformed/processed data to a destination. Configuring one or more output satisfies this requirement.
There are a number of different outputs you can choose from, and are listed here:

  • Event Hub
  • SQL Database
  • Blob Storage/Data Lake Storage Gen2
  • Table Storage
  • Service Bus Topic
  • Service Bus Queue
  • Cosmos DB
  • Power BI
  • Data Lake Storage Gen1
  • Azure Function

Additional information is available for each of these types here.

streamanalytics7
Figure 7: Stream Analytics Outputs

For our example I will be configuring an output to Cosmos DB. Adding an output of this type displays a blade that allows entry of the details of the output being created. You need to enter an alias name for the output, along with the choice of entering the Cosmos DB setting manually or selecting from your Azure subscription. We are selecting an existing Cosmos DB resource that was created previously. Please note that you must have previously created a container in Cosmos DB before specifying the Container Name and Document Id values.

streamanalytics8
Figure 8: Adding CosmosDB Output

Queries

Once you have one or more inputs and outputs configured, you can now create a query to process the incoming data. Queries are the mechanism used by Stream Analytics to transform the incoming data in real time. These queries are written using Stream Analytics Query Language which are similar to SQL statements.

Structurally the query has a SELECT statement which selects data from one or more configured inputs and uses an INTO clause to specify an output to emit the transformed data to. There are also FROM, JOIN and WHERE clauses which help specify and filter the incoming data and perform processing against it.

In our example, when you click on the Query section under Job Topology, a page is displayed that shows the current query for this Stream Analytics job. You are able to edit in the window to specify your select query statements. Notice that we are using SELECT * wildcard to specify all fields from the input source and saving those to the specified output.

streamanalytics9
Figure 9: Stream Analytics Queries

You can see preview data displayed from our input source (IoT Hub device):

streamanalytics9-1
Figure 10: Input Preview Data

This querying editor also has the ability to perform discrete field selection and filtering. So, for example if we wanted to specify certain fields we could modify our query to look like this:

streamanalytics10
Figure 11: Discrete Field Selection in Queries

Additionally, if we wanted to filter the incoming data we have that ability as well. For example, if we wanted to exclude any incoming temperature data that did not exceed 100 we could specify a filter like this:

streamanalytics11
Figure 12: Query Filtering

Once we have the inputs, outputs and queries configured we are ready to turn on our Stream Analytics job. Looking at the Overview page we can see at a glance the configured inputs, outputs and what the query is specified as.

When we start the Stream Analytics job and wait a few minutes, we should begin to see data coming in and being processed. The Monitoring and Resource Utilization graphs would show some data coming through as it is being processed.

streamanalytics12
Figure 13: Stream Analytics Graphs

We can also see our data appearing in our CosmosDB database, as shown here:

streamanalytics13
Figure 14: CosmosDB Data Records

Conclusion

In conclusion, Azure Stream Analytics provides the ability to process complex events in real time, with the purpose of providing real time analytics to incoming data sources. We also showed an example on how to create and configure Azure Stream Analytics to accept incoming IoT device telemetry data and process and save it into a CosmosDB database. I hope this article has been helpful and please click here for more information.

Cheers!

Azure IoT Hub Messaging

The Azure IoT Hub service provides the ability for bi-directional communication with your devices. You can use Azure IoT Hub to send messages from your devices to your IoT cloud solutions, as well as send commands from your IoT cloud solutions to your devices. This article will describe the different types of communications possible with Azure IoT Hub and how you can use them.

Prerequisites

Before you can send/receive messages between Azure IoT Hub and your devices you need the following:

  • An Azure account. Sign up for a free trial here
  • Provision an instance of IoT Hub
  • Create at least one device with your IoT Hub

Bi-directional communication means just that – in both directions. What that means for Azure and IoT is that you either send data from your devices to IoT Hub, or data or commands from IoT Hub to your devices. I will describe the different options to accomplish this, as seen below:

  • Device-to-Cloud messages
  • Cloud-to-Device messages
  • Device Twins
  • File Upload
  • Direct Methods

Device-to-Cloud Messages

Sending information from your IoT devices to the cloud is supported by Azure IoT Hub with device-to-cloud (D2C) messages. Your devices can send telemetry time series and alert data to IoT Hub to be routed to other services for processing.

Azure IoT Hub comes with a built-in endpoint that back-end services can use to read telemetry data being sent from your devices. This endpoint is compatible with Event Hubs and you can also use standard IoT Hub SDKs to read from this endpoint.

When D2C messages are sent from your devices to IoT Hub, by default they are routed to this built-in service endpoint (messages/events). This endpoint is exposed to your devices with MQTT, AMQP and HTTPS protocols. You also have the ability to route these messages to custom endpoints that you create.

Messages can be routed to multiple endpoints, so if you create custom endpoints and setup the routing queries you can have a message be routed to more than one endpoint. IoT Hub currently supports the following custom endpoints:

  • Azure Storage (Blob or Data Lake Gen2)
  • Service Bus Queues
  • Service Bus Topics
  • Event Hubs

Here is an example route that filters out messages that have a temperature alert flag specified and routes the messages to the default events endpoint.

Figure 1: IoT Hub Routing Example

Please note that the built-in Event Hubs endpoint (messages/events) is the fallback route, so if incoming device messages do not satisfy query conditions for any custom endpoints they will be sent to this fallback route. You must enable the fallback route capability when you turn on message routing or your incoming messages will not be routed properly.

D2C messages received are stored temporarily by IoT Hub up to 7 days. They are lost if not processed by a configured back-end service. Messages can also have a maximum size of 256KB for each message. The throughput of what an IoT Hub can handle depends on the pricing tier. Details are here.

Additional details on D2C communications can be found here.

Cloud-to-Device Messages

Sending one-way communications from a cloud solution to one or more devices is possible with cloud-to-device (C2D) messages. Your cloud solutions can send notifications to devices, or commands that can alter the state of devices (device twins).

Azure IoT Hub will send messages to devices through a service-facing endpoint (messages/devicebound), and the device(s) then receive the messages through a device-specific endpoint (devices/{deviceid}/messages/devicebound).

Azure IoT Hub provides a guarantee that messages will be sent to devices at-least-once. In order to do this, IoT Hub persists C2D messages in per-device queues. These messages persist in the queues until the devices explicitly acknowledge completion (aka – they received the message successfully). Once acknowledged, IoT Hub removes the message from the appropriate per-device queue. This approach guarantees resiliency in the case of connectivity and/or device failures.

Additional details on C2D communications can be found here.

Device Twins

JSON documents that contain device state information are what is known as device twins. These documents store the metadata, configurations and conditions for a device. Azure IoT Hub maintains a device twin for each device you connect to IoT Hub. There are multiple uses that device twins provide:

  • Store device-specific metadata in the cloud
  • Report current state information about the device
  • Synchronize the state of long-running workflows between the device and cloud solution
  • Querying device metadata, configuration or state

The structure of a device twin is shown in Figure 2:

IoTHubMessaging2
Figure 2: Device Twins

Tags are a section of the JSON document that the cloud solution can read/write to. Devices cannot see these tags or their values.

Desired Properties are a section of the JSON document used to synchronize device configurations or conditions. The cloud solution can set these properties and the device can read them. The device can also receive notifications that the desired properties have changed.

Reported Properties are a section of the JSON document also used to synchronize device configurations or conditions. Devices can set these properties and the cloud solution(s) can read them.

Root object properties define the device identity and also provide container objects for tags and the reported, desired properties sections.

Here is an example of a device twin:

{
    "deviceId": "testDevice1",
    "etag": "AAJWKSOAAAAAc=", 
    "status": "enabled",
    "statusReason": "provisioned",
    "statusUpdateTime": "0001-01-01T00:00:00",
    "connectionState": "connected",
    "lastActivityTime": "2020-02-28T12:44:48.976Z",
    "cloudToDeviceMessageCount": 0, 
    "authenticationType": "sas",
    "x509Thumbprint": {     
        "primaryThumbprint": null, 
        "secondaryThumbprint": null 
    }, 
    "version": 1, 
    "tags": {
        "$etag": "123",
        "deploymentLocation": {
            "building": "A",
            "floor": "1"
        }
    },
    "properties": {
        "desired": {
            "telemetryConfig": {
                "sendFrequency": "5m"
            },
            "$metadata" : {...},
            "$version": 1
        },
        "reported": {
            "telemetryConfig": {
                "sendFrequency": "5m",
                "status": "success"
            },
            "batteryLevel": 55,
            "$metadata" : {...},
            "$version": 4
        }
    }
}

The cloud solution backend can operate on the device twin using these operations (through HTTPS):

  • Retrieve Device Twin: Returns the JSON document for the device twin
  • Update Device Twin: Enables the partial update of the JSON document. Tags and Desired Properties can only be updated. Cloud solution can also add or remove properties in these sections. Setting a property to null removes it from the section. There is also the ability to completely overwrite the Tags and Existing Properties with new JSON.
  • Receive Device Twin Notifications: Allows the cloud solution to be notified when a change is made to the device twin. A route in the IoT cloud solution called twinChangeEvents must be created and configured as a route for this operation to succeed.

The device can operate on its device twin using these operations:

  • Retrieve Device Twin: Returns the JSON document for the device twin. Tags are not visible.
  • Update Device Twin: Enables the partial update of the JSON document. Reported Properties can only be updated. This operation works in the same way as the cloud solution updates a device twin.
  • Observe Desired Properties: The device can choose to be notified of updates to the Desired Properties when they occur.

There are size limitations on the different sections in the device twin, as seen here:

  • Tags: 8KB max size on values
  • Desired Properties: 32KB max size on each value
  • Reported Properties: 32KB max size on each value

As an example, if the cloud solution wants to change the sendFrequency value for the device it modifies the device twin value for sendFrequency, as seen here:

"desired": {
   "telemetryConfig": {
   "sendFrequency": "5m"
},
…

The device(s) would be notified of this change immediately or when they become connected (if offline). The device would then report back to the cloud solution the updated configuration on the device:

"reported": {
   "telemetryConfig": {
   "sendFrequency": "5m",
   "status": "success"
},
…

File Upload

IoT devices can start file uploads through a device-facing endpoint in IoT Hub. You can have your devices request file uploads to an associated Azure storage account in the cloud, with IoT Hub acting as a dispatcher for the uploaded file (requested by a device) to the Azure storage account. File uploads can be used for large files such as media files or large telemetry batches.

In order for file uploads to work, you must first associate an Azure storage account to IoT Hub. This can be done via the Azure Portal or programmatically through the IoT Hub REST APIs.

IoTHubMessaging3
Figure 3: File Upload Section

Figure 4: File Upload Configuration in IoT Hub

Once you have made this association you can write code in your device to upload files to the cloud. IoT Hub will handle the upload to its associated blob storage account and provide notifications that you can monitor. Here is a great tutorial that reviews the steps needed to perform a file upload from your device.

Direct Methods

Direct methods provide an alternative way of communication with devices that work independently of the other communication options. Direct methods allow us to call a method on the device as a local service, which means we can get immediate feedback from the results of that call.

Direct methods are implemented on the device and use a request-response pattern when communicating. You can invoke a direct method through a service-facing URI and are a synchronous call. The invocation will either succeed or fail (after the timeout period), returning a result with an optional payload. The benefits/drawbacks of direct methods are as follows:

Benefits

  • Define multiple named direct methods
  • Pass an optional payload
    Returns a status code and optional payload
  • High throughput

Drawbacks

  • Only supports AMQP or MQTT
  • Doesn’t work if device is offline

As an example, here is how you would write code on your device to receive calls to a direct method:

await device.SetMethodHandlerAsync(“showMessage”, ShowMessage, null);
…


private static Task ShowMessage(MethodRequest methodRequest, object userContext)
{
…
}

Calling the SetMethodHandlerAsync class registers a named method handler with the device, which in this example the method name is ShowMessage. Parameters are the name of the message handler method as well as the reference to that handler method and an optional context object.
The definition of the ShowMessage handler method is also defined, which takes a MethodRequest object parameter and optional context object and returns a Task object.

Now that our devices are able to handle direct messages from the cloud, the only other thing to do is write code in our IoT cloud solution to call the direct method on our device. We can do this by defining a method to invoke the direct method call on our device, as seen in this console application example:

static async Task Main(string[] args)
{
   …
   var serviceClient = ServiceClient.CreateFromConnectionString(svcConnString);
   await CallDirectMethod(serviceClient, deviceId);
   …
}


private static async Task CallDirectMethod(ServiceClient sClient, string deviceId)
{
   var method = new CloudToDeviceMethod(“showMessage”);
   method.SetPayloadJson(“’This is a test direct method’”);

   var response = await sClient.InvokeDeviceMethodAsync(deviceId, method);

   Console.WriteLine($”Response status: {response.Status}, payload:{response.GetPayloadAsJson()}”);
}

As you can see, the Main method creates an instance to our IoT hub using the ServiceClient class. We then create a private method named CallDirectMethod(), which takes the instance of our ServiceClient class along with the Device ID value. This method instantiates the direct method via the CloudToDevice() class, which we specify the name of the direct message when instantiating (showMessage in our example). We can then set the payload JSON and asynchronously invoke the direct method on the device. A response is returned to let us know if the method ran successfully or not.

Conclusion

In conclusion, we described the different types of communications possible with Azure IoT Hub and how you can use them. I hope this article has helped you understand the different types of Azure IoT Hub messaging.

Cheers!

Azure IoT Central

Azure IoT Central is Microsoft’s SaaS (software-as-a-service) offering that allows you to quickly provision an IoT solution. This service reduces the burden and cost of developing, managing and maintaining enterprise-level IoT solutions, allowing you to focus more on your IoT data and how it can transform your business. I will be going through the steps to create an Azure IoT Central application and how easy it is to create and maintain an IoT solution.

You can use IoT Central to create a custom, cloud-based IoT solution without having to worry about the details of the underlying infrastructure. Customization of your solution can all be done from the browser in the application UI, and includes capabilities to:

  • Define the layout of properties and settings on a device template
  • Configure custom dashboards
  • Configure custom analytics to explore device data

 

Prerequisites

The only prerequisites needed are a Microsoft account (personal, work or school) as the creation process will ask you to login to your account, and an Azure subscription.

 

Getting Started

First you need to navigate to the Azure IoT Central Build site. You will notice a left-hand navigation menu which consists of several sections:

IoTCentral1

Figure 1: Azure IoT Central Menu

 

Clicking on the Build menu selection will take you to a page that displays a number of options to build your IoT application. You can start with a number of pre-generated templates that are geared towards specific industries or start with a blank application template.

Once you select either an industry-specific or custom template, a page displays that allows you to specify information about your application.

IoTCentral2

Figure 2: Creating an IoT Central Solution

The Application Name is pre-populated for you, as well as the URL. Please note that the application name and URL must be globally unique. The Application Template dropdown is only visible on custom applications, as you’re already selecting a specific template if you choose one of the industry-specific templates.

You also need to choose the pricing plan – you can try the solution for free for 7 days or select one of the other pricing tiers. These other tiers are based on how often you want to send messages from your devices. The Billing Info allows you to select the Directory (Azure Active Directory) and Azure Subscription you wish to have this solution hosted from. Location is also required as the system will provision the IoT Central solution resources in the closest region.

 

Web Application

Once the system provisions your IoT Central application you are taken to the Dashboard page, which displays some starting links such as adding devices, quick start demos, tutorials and documentation.

IoTCentral3

Figure 3: Dashboard page

 

Devices

Adding devices is easy in IoT Central – just click on the Devices tab and you are taken to a page that displays all the devices currently in the application. Adding a device is done by clicking on the +New button. A dialog window is displayed allowing you to configure the details about the new device being added:

IoTCentral4

Figure 4: Adding a new device

Device ID is a unique identifier that is used by the device to connect, while Device Name is a friendly name for the device. You can also toggle the Simulated option which specifies whether your device is simulated or real.

 

Device Templates

You can also create one or more device templates, which can then be used to create new devices that have the template characteristics. Click on the +New button on the Device Templates tab (under App Settings) and you can select the template type:

IoTCentral5

Figure 5: Device Templates

You have the ability to create a template for an IoT device, Azure IoT Edge device, or one of the pre-configured device templates listed. The list of certified devices (with IoT Plug and Play) are shown here and more are being added all the time.

Once you publish your device template, you can add devices based on that template. On the Devices tab you will notice a selection with the name of your device template that you can click on. This will display a list of the current devices that are based on this device template. Clicking the +New button will allow you to create a device based on this template.

IoTCentral6

Figure 6: Creating a Device using Device Template

 

Device Groups

Device Groups are a logical grouping of devices in the system based on specific attributes. You can create one or more Device Groups to filter one or more devices in your system. This is useful if you have hundreds (or thousands) of devices and you need to look at specific devices quickly.

Navigate to the Device Groups tab and click on the +New button to create a new device group. You will see a page that allows you to enter information for your new device group such as the Name and Scope. Adding one or more filters in the Scope area will define the filters needed to only display the device(s) you want to see.

IoTCentral7

Figure 7: Device Groups

 

Rules

You can create telemetry-based rules in IoT Central. This is done under the Rules section in the application. Clicking on the Rules tab will display a list of the current rules in the system. Create a new rule by clicking on the +New button. This will display a page that allows you to create a new rule.

IoTCentral8

Figure 8: Rules

Give the rule a name and then specify a target device template, which will define the devices to which the rule applies. If you need to specify less than all the devices in the device template you select, create one or more filters to filter the list of devices further.

Conditions are created to specify when the rule is triggered. You can specify one or more conditions the rule must meet to be triggered.

There is also a section for Actions. These are the things that happen once the rule has been triggered. You notice you can either send an email or call a web hook when the rule is triggered. You can create one or more actions if you choose.

IoTCentral9

Figure 9: Email Action

 

IoTCentral10

Figure 10: Webhook Action

 

Analytics

Azure IoT Central also has the capability to analyze device data including time series information. You can specify the telemetry values to display along with splitting the data collections by attributes (such as Device ID). Select the Device Group and Telemetry value(s) to analyze and click the Analyze button when you are finished.

IoTCentral11

Figure 11: Analytics

 

Jobs

You can use Azure IoT Central to manage your connected devices using jobs. Jobs are useful for performing bulk updates to your device properties or for running commands. Clicking on the Jobs tab will display a list of the existing jobs in the system, job status and when they were last run. Creating a new job will allow you to specify details of the job such as Name, what device group the job targets as well as the Job Type and detailed information.

IoTCentral12

Figure 12: Creating a Job

Running a job is done by clicking on the Run button. This will execute the job and return results of the operation.

IoTCentral13

Figure 13: Running a Job

 

Administration

The IoT Central application also allows administration of the website, including Users in the system, Roles, Pricing and the Device Connections. You also have the ability to permanently delete the solution.

IoTCentral14

Figure 14: Administration

 

Underlying Infrastructure

Since IoT Central is a SaaS (software-as-a-solution) offering, you do not have access to the underlying architectural components in Azure. What this means is that Microsoft fully hosts and supports the IoT Central application but they also manage the resources themselves. You do not have the ability to configure/customize/remove any of the individual Azure resources used in your IoT Central application. If you need to customize the IoT Central application beyond what is provided to you through the SaaS offering, then you need to consider Azure IoT Solution Accelerators which do provide the ability to manage and modify the underlying Azure components.

 

In conclusion, Azure IoT Central is a nice offering from Microsoft if you want to get your IoT solutions up and running quickly! Device provisioning and customizations are part of this service, as well as Rules, Jobs, and analytics for the device data. I hope this brief tour of Azure IoT Central has been helpful to you.

Cheers!

Message Routing with Azure IoT Hub

IoT solutions can be complex, especially if you have multiple devices that you need to send to more than one destination in your cloud solution. Azure IoT Hub provides the capability to send your device data to multiple destinations through message routing. This article will describe the available options with message routing and how they are used.

IoT solutions can be complex, especially if you have multiple devices that you need to send to more than one destination in your cloud solution. Azure IoT Hub provides the capability to send your device data to multiple destinations through message routing. This article will describe the available options with message routing and how they are used.

Prerequisites

Before you can use message routing with your devices you will need the following:

  • An Azure account. Sign up for a free trial here
  • Provision an instance of IoT Hub
  • Create at least one device with your IoT Hub

Incoming Messages

The devices in your IoT solution will be sending telemetry data into Azure IoT Hub. Conceptually, you normally think of these messages all being sent to one place and being handled/processed by the hub. However, what if you want these messages sent to different cloud services that may be part of your solution? In the past you might have had custom code to handle these messages and route them appropriately. That made your solution overly complex when you were required to handle all the details about these incoming messages, such as handling transient faults, how to deal with lost messages, and maintaining high reliability.
Fortunately, Microsoft has made this easier for us by providing message routing as part of Azure IoT Hub. IoT Hub will now take care of all those complex details for routing messages so we can focus on our custom IoT solution code. IoT Hub provides scalability for incoming messages, handling transient faults, and how to handle lost messages. The details of message routing are abstracted from us, and we only need to know how to configure the routing to specify where we want our messages to go.
MessageRouting1

Figure 1: Message Routing

 

Endpoints

There are a number of places you can route your device messages to by configuring IoT Hub. These are the final destinations of your data, allowing backend processing services to execute actions against the data you route to it. The different types of endpoints are as follows:

  • Event Hub
  • Service Bus Queue
  • Service Bus Topic
  • Storage

Also note that paid SKU hubs can have up to 10 additional endpoints while free SKU hubs can only have 1 additional endpoint.

Event Hub

Creating an event hub allows you to route your device data to a specific event hub resource to allow further/dedicated processing. Note that IoT Hub comes with a built-in endpoint that is compatible with this type. Here is an example of adding a new event hub endpoint to IoT Hub:

MessageRouting2

Figure 2: Event Hub Endpoint

 

Service Bus Queue/Topic

You can create a custom endpoint to a Service Bus Queue/Topic, which would then create a record that can be processed by an Azure function for example. Here is an example of adding a new Service Bus Queue endpoint to IoT Hub:

MessageRouting3

Figure 3: Service Bus Endpoint

Creating a custom endpoint to a Service Bus Topic works in the same way you would create one for a Service Bus Queue, but please note that your Service Bus Namespace must be the Standard (or higher) pricing tier for topics to be available. Also be aware that you cannot create a custom endpoint to a Service Bus Queue/Topic if they have sessions or duplicate detection enabled.

Storage

You can also create a custom endpoint that uses Azure blob storage, allowing the saving of incoming messages to an existing container (as blobs only). Here is an example of adding a new Azure Storage endpoint to IoT Hub:

MessageRouting4

Figure 4: Storage Endpoint

Batch Frequency specifies (in seconds) the maximum amount of time before the system will batch and write records to Azure Storage. The Chunk Size Window specifies the maximum size of blobs (in MB). Apache AVRO is the default encoding, although you can specify JSON.

Routes

Routes are created/configured to direct the appropriate data source to an endpoint for processing. When you create an endpoint you must specify the Name, Endpoint (built-in or custom), Data Source, and filter query.

MessageRouting5

Figure 5: Adding a Route

The Data Source field specifies where the data is coming from and must be one of the following selections:

MessageRouting6

 

You also have the ability to filter the incoming message data, which can be done using queries in the Routes. IoT Hub defines a common format for all device-to-cloud messaging, and filtering is done by writing a query against this common format. Here is an example:

{ 
  "message": { 
    "systemProperties": { 
      "contentType": "application/json", 
      "contentEncoding": "UTF-8", 
      "iothub-message-source": "deviceMessages", 
      "iothub-enqueuedtime": "2017-05-08T18:55:31.8514657Z" 
    }, 
    "appProperties": { 
      "processingPath": "{cold | warm | hot}", 
      "verbose": "{true, false}", 
      "severity": 1-5, 
      "testDevice": "{true | false}" 
    }, 
    "body": "{\"Weather\":{\"Temperature\":50}}" 
  } 
}

Querying on message system properties should prefix a ‘$’ symbol on the query. Queries on application properties do not need this symbol prefixed. For example, querying on the system property ‘contentEncoding’ would look like this:

$contentEncoding = 'UTF-8'

Querying on an application property would look like this:

processingPath = 'hot'

Please note that device-to-cloud messaging is not case-sensitive, so using unique property names is recommended. Here is an example of a query that filters records that have the application property ‘temperatureAlert’ = true.

MessageRouting7

Figure 6: Message Routing Queries

A fallback route is present so that any messages that do not satisfy any specified query conditions will be sent to this route. The default fallback route is the ‘messages/events’ route that is used by the built-in endpoint (compatible with Event Hub). You can turn on message routing in IoT Hub, which will allow you to create custom routes. Once a custom route is created, data stops flowing to the built-in endpoint unless you specifically create a route to that endpoint. Please be aware that if you delete all custom routes and you disable the fallback route, data will not be sent to any destination.

 

In addition to telemetry data, routes can be created for other events such as Device Twin changes, Device Lifecycle events and Digital Twin changes (currently in public preview). If you create a route with the data source set to one of these other events, IoT Hub will send messages to the appropriate endpoint for that event (Device Twin, Digital Twin, Device Lifecycle).

 

Conclusion

In conclusion, using message routing with Azure IoT Hub increases the flexibility, reliability and scalability of your IoT solution. Routing also enables you to keep track of specific events in your solution, such as device twins and device lifecycle. I hope this article has helped you understand the different options for message routing with Azure IoT Hub.

Cheers!

Creating and Configuring Azure IoT Hub

Microsoft provides the Azure IoT Hub service to assist in implementing Internet of Things (IoT) solutions. This fully managed Azure service enables reliable and secure bidirectional communication between your IoT devices and a back-end solution hosted in Azure. I am going to review the steps to create an IoT hub in Azure, along with how to register a device and configure tier and scaling options.

Prerequisites

The only prerequisite you need to follow along is an Azure account. You can sign up for a free trial here.

Creating an IoT Hub Instance

Azure Portal

Probably the easiest way to create an instance of IoT Hub is to use the Azure portal. You do this by first logging into the Azure portal with your Azure credentials, and then clicking on Create a Resource in the upper left. This displays a new blade with a number of choices under Azure Marketplace. Click on Internet of Things and you should see the IoT Hub selection, as seen here:

CfgIoTHub2

Clicking on IoT Hub will display a new blade that allows you to enter in details of your new IoT Hub, such as the Subscription, Resource Group, Region and IoT Hub name. The Subscription is auto filled with the subscription you are currently logged into. You can select a Resource Group to place the new IoT Hub in or create a new one. Region specifies the location that your IoT Hub will be created in, and the IoT Hub Name is a globally unique name you give to your IoT Hub.

CfgIoTHub3

Clicking Next: Size and Scale takes you to the next screen which allows you to select a pricing tier for your IoT Hub, along with the number of IoT Hub units allocated. The pricing tiers include a Free Tier as well as 3 levels of either Basic or Standard Tiers (B1-3, S1-3 respectively). The choice you make here determine a number of options relating mainly to data throughput and scaling. Details are here.

At the time of this writing, the highest pricing tier selection is S3: Standard Tier as shown below:

CfgIoTHub4

Clicking Review + create will take you to a summary page listing out the selection choices you made. Clicking Create here will instruct the Azure Portal to create the IoT Hub with the choices you selected.

CfgIoTHub5

Azure CLI

Another way to create an instance of IoT Hub is by using the command-line interface (CLI). Azure hosts an interactive shell environment called Azure Cloud Shell that you can use to execute CLI commands. You can launch Azure Cloud Shell by either going to https://shell.azure.com in your browser, or by clicking on the Cloud Shell button in the upper right in the Azure Portal.

CfgIoTHub9

Once you are in your Cloud Shell environment, you can execute the following command to create an instance of IoT Hub:

az iot hub create --name {your iot hub name} --resource-group {your resource group name} --sku S1

The only required parameters are the name of your IoT Hub along with the resource group it will be associated with. Insert a globally unique name for your IoT Hub along with the name of an existing resource group. The optional –sku parameter designates what pricing tier to create the IoT Hub with. Pricing details are located here.

There are additional optional parameters when creating an IoT Hub and are listed here.

Configuring IoT Hub Tiers

There are two tiers for Azure IoT Hub: Basic and Standard. You should consider what your IoT solution is designed to do before you select a tier. The difference between Basic and Standard tiers are the number of features they support. A Free tier is also provided, which has all the functionality of the Standard tier but has limited messaging allowances.

If your solution collects data and you are analyzing it centrally, then the Basic tier is probably the best choice. If you want to use advanced configurations to control your IoT devices remotely, or you want to distribute some of your workloads to the devices themselves, then the Standard tier is probably the best choice.

Each tier has three levels, listed in numerical order (1, 2, 3). The difference between these three levels is how much data they allow to be send from devices per day:

  • B1, S1 tiers: 400,000 messages/day/unit
  • B2, S2 tiers: 6 million messages/day/unit
  • B3/S3 tiers: 300 million messages/day/unit

The following features are available in all IoT Hub tiers:

  • Device-to-cloud telemetry
  • Per-device identity
  • Message routing, enrichments and Event Grid integration
  • HTTP, AMQP, and MQTT protocols
  • Device Provisioning Service
  • Monitoring and diagnostics

The following features are only available in the Free/Standard IoT Hub tier:

  • Cloud-to-device messaging
  • Device twins, Module twins, and Device Management
  • Device streams (preview at the time of this blog)
  • Azure IoT Edge
  • IoT Plug and Play (preview at the time of this blog)

Registering Devices

Once you have your IoT Hub instance created, you can register one or more devices identities that you intend to use with your IoT Hub.

Azure Portal

After you log in to the Azure Portal, navigate to your IoT Hub and click on the IoT Devices selection (under Explorers section):

CfgIoTHub6

This will display a list of the current devices created for the IoT Hub. You can click on the + New link towards the top of the page and a new blade will be displayed, allowing you to enter the following information about your device:

  • Device ID – enter a name for the device. This is used for authentication and access control
  • Authentication Type – select how the device will authenticate to IoT Hub. Symmetric Key is the default and allows entry of the primary/secondary keys (see below). X.509 Self-Signed and X.509 CA Signed allow you to select security certificates for the device (certificate created prior)
  • Primary/Secondary Keys – allows you to set a unique value for accessing the device. Disabled if auto-generate keys is checked (valid for Symmetric Key authentication type only)
  • Auto-generate keys – this will instruct Azure to generate unique values automatically for the primary/secondary keys (valid for Symmetric Key authentication type only)
  • Connect this device to an IoT Hub – specifies if this device will interact with this IoT Hub
  • Parent Device – specify if this device has a parent device. Used for IoT Edge

CfgIoTHub7

Clicking Save will instruct the Azure Portal to create the device with your selections. Once completed, you can access the device details by clicking on its name in the list of devices:

CfgIoTHub8

Azure CLI

You can also register device identities with your IoT Hub using Azure CLI. Execute the following command in your Cloud Shell instance, replacing {iothub_name} with your IoT Hub name and {device_id} with the name you wish to use for your device:

az iot hub device-identity create -n {iothub_name} -d {device_id}

The console will emit a JSON string upon success which give information about the device you just registered, such as primary/secondary key values, its connection state and whether it has IoT Edge capabilities:

CfgIoTHub10

You can verify in the Azure portal that the device is registered by navigating to the IoT Devices section in your IoT Hub. Your newly-registered device should be displayed in the list:

CfgIoTHub11

Additional parameters can be found here.

I hope this article has helped you understand what Azure IoT Hub is and how to create an instance and register your IoT devices to it, so you can begin sending data to Azure.

Cheers!

Beginning IoT – Installing Windows 10 IoT Core on an x86/x64 Device

This is the second part in a 2-part series on how to install Microsoft Windows 10 IoT Core on an Internet-of-Things (IoT) device. Part 1 described how to install Windows 10 IoT Core on a Raspberry Pi 3 device. This article will focus on the steps required to install Windows 10 IoT Core on an x86/x64 device.

 

What is Windows 10 IoT Core?

Windows 10 IoT Core is a version of Microsoft’s Windows 10 operating system that has been optimized for smaller devices that can run on either ARM or x86/x64 devices. These devices can run with or without a display device.

When we talk about the different IoT devices, the processor type needs some explanation. ARM devices are called Advanced RISC Machines, with RISC standing for Reduced Instruction Set Computer. What this means is that the processor has been slimmed down to only include a reduced set of commands it can process. While this means that the processor can’t do certain things, it requires a low amount of power to execute what it can do, so that translates to increased battery life. The Raspberry Pi is classified as an ARM device.

Devices with the x86/x64 architecture are classified as CISC processors, which stands for Complex Instruction Set Computer. These processors do not have their instruction sets slimmed down, so they can perform more complex operations, at the cost of increased power consumption (and therefore lower battery life). Intel’s Baytrail devices running the Intel Atom processor E3800 is an example of an x64 device.

https://www.intel.com/content/www/us/en/embedded/products/bay-trail/overview.html

 

Prerequisites

Before you can install on an x86/x64 device, you need to make sure you have a PC that is running Windows 10 1507 (version 10.0.10240) or higher. You can find out what version you are running by clicking on the search box (next to the Start button) and typing ‘winver’. This will display a dialog as shown here:

1

 

For certain x86/x64 device you can use IoT Core Dashboard to run through the installation process. However, since the process is similar using IoT Core Dashboard (which I already covered in Part 1), I am going to go through the installation process steps using the Windows ADK IoT Core Add-Ons.

You will need an IoT device to install on – here are some options that Microsoft supports:

https://docs.microsoft.com/en-us/windows/iot-core/learn-about-hardware/socsandcustomboards

 

We will be using the Intel Atom E3800 (aka Baytrail) built on an industrial PC for examples in this article. Typically we would need a micro SD card for storage but since the industrial PC comes with onboard memory storage, we do not need an SD card.

 

For software, we will need to install the following on the PC we build the image on:

 

Installation Steps

Once you have the prerequisites installed, you are now ready to begin the installation process. We will be building a basic image, which involves combining the Windows IoT Core packages along with a board support package for the target hardware (Baytrail device) into a flashable file (FFU file).

First off, you need to set your OEM name, which will help you distinguish your created packages from packages other manufacturers have created. Edit the setOEM.cmd file, located at C:\IoT-ADK-AddonKit and set the OEM_NAME variable accordingly. Please note you can only use alphanumeric characters.

4

 

You should now open the IoTCoreShell.cmd file, which is a specialized command-line window that you will be doing a lot of the work for building the image. This file is located in the directory where you installed the IoT Core ADK Add-Ons (C:\IoT-ADK_AddonKit). Open an administrator-elevated privilege command window and navigate to the C:\ IoT-ADK_AddonKit directory. Type in IoTCoreShell.cmd to open the IoT Core Shell. This application will prompt you to select the architecture you are working with (1 for ARM, 2 for x86, 3 for x64). We are creating an x64 image so select x64.

5

 

At this point, you need to install certificates which will be used to sign the package binaries. Since this article is focused on a test image, you can run the installoemcerts.cmd command to install test certificates in the root certificate store of the PC you are building the image on. This only needs to be done the first time you are building an image.

6

 

The next step is to extract the board support package (BSP) files for the device you are building an image for, and run the buildpkg.cmd command to build the package files used in creating the image. You can download and extract this BSP zipfile and copy to C:\IoT-ADK-AddonKit\Source-x64\BSP directory to begin using for building an image.

7

 

You can now begin to create the packages and build the FFU image for the Baytrail x64 device. Go back to the IoTCoreShell.cmd command window and enter buildpkg all which will build the .cab files for all the BSP directories the program sees under C:\IoT-ADK-AddonKit\Source-x64\BSP. Please note that if you had selected x86 when you ran the IoTCoreShell.cmd, running this command to build all the packages would look in the C:\IoT-ADK-AddonKit\Source-x86\BSP to build any BSP files located there.

8

 

Once the program finishes building all the BSP packages, you can now create a new project by entering the following, where Product_Name and BSP_Name are the name of the product you would like and the BSP name, respectively.

newproduct <Product_Name> <BSP_Name>

So, for example, entering newproduct MyBayTrailDevice BYTx64 will create a project and its files under C:\IoT-ADK-AddonKit\Build\amd64\MyBayTrailDevice for the BYTx64 board support package files.

9

10

You are now ready to build the actual FFU flashable image file. This can be done by entering buildimage <Product_Name> Test, replacing Product_Name with your product name (MyBayTrailDevice in our example). The second parameter specifies whether you are building a Test or Retail image. This process takes about 20-30 minutes to complete and once finished you will have a file named flash.ffu created under the C:\IoT-ADK-AddonKit\Build\amd64\ MyBayTrailDevice\Test subdirectory.

11

 

If you encounter any errors, the buildimage process will error out and specify a log file that has detailed information on the error.

Now that you have a flashable FFU image file, you will need to flash it onto the IoT device you are working with. For our example, this is the Baytrail device and since it has onboard storage space we need to use a bootable USB thumbdrive with the FFU file on it. In order to create this, we will use Windows PE to create a bootable disk and then copy the flash.ffu file onto it. Here are instructions to create a bootable WinPE thumbdrive:

https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/winpe-create-usb-bootable-drive

 

Copy the flash.ffu file to the root of this bootable drive once you’ve created it. You are now ready to insert this USB thumbdrive to the IoT device and power it up. Make sure you specify in the IoT device’s BIOS to boot first from a USB drive.

12

WinPE will boot up and open a command window for you at the x: drive. Change to the d: drive and enter dir to see your flash.ffu file. WinPE comes with DISM, the Deployment Image Servicing and Management tool and we will be using this to flash the FFU file onto the IoT device. Enter the following in the command line to flash the FFU file:

dism.exe /apply-image /ImageFile:Flash.ffu /ApplyDrive:\\.\PhysicalDrive0 /skipplatformcheck

13-14

 

Once DISM has successfully completed the flashing process, you can power down the IoT device and remove the USB thumbdrive. Turn on the IoT device and have it boot normally off its storage. After a few minutes you should see the Windows 10 IoT Core startup screen and it should prompt you to select a language and whether you want Cortana activated. Once you make these selections the default application will appear.

15

 

Congratulations! You have successfully installed Windows 10 IoT Core on a x64 IoT device! Future steps can now be to modify the image to have it include your custom application or you can add drivers to the image if you need other functionality (such as Bluetooth or serial communications).

 

Enjoy!

 

Beginning IoT – Installing Windows 10 IoT Core on a Raspberry Pi

This is the first part in a 2-part series on how to install Microsoft Windows 10 IoT Core on an Internet-of-Things (IoT) device. This article will focus on the steps required to install Windows 10 IoT Core on a Raspberry Pi 3. Part 2 will focus on installation on an x86/x64 device.

 

What is Windows 10 IoT Core?

Windows 10 IoT Core is a version of Microsoft’s Windows 10 operating system that has been optimized for smaller devices that can run on either ARM or x86/x64 devices. These devices can run with or without a display device.

When we talk about the different IoT devices, the processor type needs some explanation. ARM devices are called Advanced RISC Machines, with RISC standing for Reduced Instruction Set Computer. What this means is that the processor has been slimmed down to only include a reduced set of commands it can process. While this means that the processor can’t do certain things, it requires a low amount of power to execute what it can do, so that translates to increased battery life. The Raspberry Pi is classified as an ARM device.

Devices with the x86/x64 architecture are classified as CISC processors, which stands for Complex Instruction Set Computer. These processors do not have their instruction sets slimmed down, so they can perform more complex operations, at the cost of increased power consumption (and therefore lower battery life). Intel’s Baytrail devices running the Intel Atom processor E3800 is an example of an x64 device.

https://www.intel.com/content/www/us/en/embedded/products/bay-trail/overview.html

 

Prerequisites

Before you can install on the Raspberry Pi, you need to make sure you have a PC that is running Windows 10 1507 (version 10.0.10240) or higher. You can find out what version you are running by clicking on the search box (next to the Start button) and typing ‘winver’. This will display a dialog as shown here:

1

 

You will also need to download and install the Windows 10 IoT Core Dashboard from here.

Of course you will also need a Raspberry Pi device to install onto. There are several different kits available on Amazon – I used the Canakit Starter Kit here.

Finally, you will need an SD card reader so that you can write the installation files to the SD card that will be placed in the Raspberry Pi.

 

Installation Steps

Once you have the prerequisites, you are now ready to begin the installation process.

First off, run the Windows 10 IoT Core Dashboard program, and click on Set up a new device from the menu on the left. This will display a screen that allows you to select the Device Type, OS Build and other information to configure as part of the installation.

2

Select Broadcomm [Raspberry Pi 2 & 3] as the Device Type, and Windows 10 IoT Core (17134) as the OS Build. You can also select Windows Insider Preview or Custom if you want to install a preview build of Windows 10 IoT Core, or a custom image file (flash.ffu file).

Next, insert your SD card into the Windows 10 PC you are using. Be aware that your SD card should be at least 8GB in size and I prefer formatting it prior to this step (this is optional – understand that the installation process will overwrite any pre-existing data on the SD card). The IoT Core Dashboard program should recognize the SD card and display it in the Drive selection.

You can then enter values for the Device Name and Password, and whether you want to use a Wi-Fi Network Connection when the Raspberry Pi starts up with our installation.

3

Check the box to accept the software license term and click on the Download and Install button. This will begin the installation process. During this process, Windows 10 IoT Core will be downloaded and the installation process will flash the files to the SD card.

5

If you see a prompt for UAC (User Access Control), click Yes to continue. The process may then open a command window to clean the SD card (if you had something on it previously) before it flashes the new installation onto it.

6

The installation process will then run the DISM program (Deployment Image Servicing and Management tool) to flash the installation files onto your SD card.

7

Once this is complete, the command window will close, and IoT Core Dashboard will state that your SD card is ready to be placed in the Raspberry Pi and started up.

8

Eject the SD card from your PC and place it in your Raspberry Pi. Connect an HDMI cable to a display source (monitor) and then plug in the power to start the device. If you didn’t choose to use a Wi-Fi network connection on startup you will need to plug in an Ethernet cable if you want Internet access.

IMG_0246

IMG_0243

You will first see the Windows logo with a spinner when the device is first powered on:

IMG_0236

IMG_0237

Let the device power itself on, it usually takes a minute or two (and might reboot itself). Once you get to the following screen, plugin a USB mouse and make your language selection.

IMG_0238

Clicking Next will display a screen asking if you want to configure Cortana – I selected Maybe Later.

IMG_0241

Windows 10 IoT Core will then run the default application, which looks like this:

IMG_0242

Congratulations! You have now completed the installation process and you have a standard Windows 10 IoT Core installation on your Raspberry Pi! You are now ready to begin deploying your applications to this device!

Happy coding!