Using Azure Stream Analytics with IoT Devices

Azure Stream Analytics is Microsoft’s PaaS (platform-as-a-service) event-processing engine that allows you to analyze and process large volumes of streaming data from multiple incoming sources. You can configure different input sources including IoT devices, sensors or business applications for data ingestion. Delivery outputs can also be configured to send the processed data to those destinations for performing actions or further analytics.

Overview

Azure Stream Analytics works on the concept of jobs. These jobs consist of one or more inputs, a query and an output. Stream Analytics ingests incoming data from one or more of these configured inputs and then a query is applied to this data to filter, sort, aggregate or join other reference data over a period of time. This transformed data is then sent to the configured output for storage or further downstream processing.

In this article, we will be using an example where I have a physical IoT Device (an MXChip dev kit) sending temperature data for processing.

Resource Creation

You can create an instance of Stream Analytics in the Azure Portal. Login to your portal at https://portal.azure.com. Once authenticated, click on Create a Resource to start the creation process.

streamanalytics1
Figure 1: Azure Services

Enter Stream Analytics Job in the search box and press Enter or click the magnifying glass.

streamanalytics2
Figure 2: Creating Stream Analytics Job Resource

This will return a list of results that match your entry. Select Stream Analytics Job to create a new instance of Stream Analytics. A description page will appear describing what Stream Analytics is, along with available pricing plans and documentation resource links. Click Create to begin the creation process.

streamanalytics3
Figure 3: Stream Analytics Job Overview

You need to provide some details when creating a Stream Analytics job in Azure. First, you will need to specify a name for your Stream Analytics instance. This name only needs to be unique to your subscription.

There is also a dropdown for selecting the Azure subscription to create the Stream Analytics instance under. It is auto populated with your current subscription but if you have multiple subscriptions you can select from one in the list.

You also need to specify a resource group that the Stream Analytics instance is hosted under. You can either select from a list of existing resource groups or click Create New to create a new resource group immediately and use that for your Stream Analytics instance.

Location specifies where you will be creating the Stream Analytics instance. Choose the location nearest to your physical location for best performance.
Hosting environment specifies where the Stream Analytics job will be deployed. Cloud specifies that the job will be hosted in the cloud (aka – the Azure Portal) while Edge specifies an on-premise IoT Edge gateway device.

Streaming Units determine the number of computational resources the Stream Analytics job will use when processing a query. Since the jobs perform all processing in memory, increasing this value will increase the amount of CPU and memory resources to process queries. This can be useful for scaling up and providing low latency stream processing.

streamanalytics4
Figure 4: Create Stream Analytics Job Details

Once created, your Stream Analytics job is stopped by default. In order to configure your job for processing you need to create three (3) things:

  • Inputs
  • Queries
  • Outputs

Let’s review each of these in detail.

Inputs

The first thing to configure are inputs. Inputs define a connection to an existing data source, and Stream Analytics accepts incoming data from these inputs. You can choose from three different input types: Azure Event Hubs, Azure IoT Hub, or Azure Blob Storage.

You can navigate to the Inputs page by clicking Inputs under the Job Topology section (on the left). This page will display a list of configured inputs while also allowing you to create a new input.

streamanalytics5
Figure 5: Stream Analytics Inputs

There are also two types of inputs: Stream and Reference. Stream inputs define an unbounded sequence of data events over time. This is the stream of input data coming from an input source and Stream Analytics requires at least one of these input types to be defined.

Reference inputs define a static set of data or data that slowly changes. This input type is typically used for data lookups or data correlation and is optional depending on your needs. An example would be having a lookup table that you would join with data from your stream input. Currently Azure Blob Storage and Azure SQL Database are supported as input sources for reference data.

For our example, we will be connecting a stream input of type IoT Hub to an existing Azure IoT Hub instance. You are prompted for an input alias name for this input source, along with selection of the subscription and IoT Hub you are specifying. There are also entries to specify IoT Hub routing endpoint to use as well as event serialization format and encoding.

streamanalytics6
Figure 6: Adding IoT Hub Input

Clicking Save will create the input and run a connection test to verify that Stream Analytics can connect to the input properly.

Outputs

Outputs must also be configured in order for the Stream Analytics job to function properly. Stream Analytics needs to send the transformed/processed data to a destination. Configuring one or more output satisfies this requirement.
There are a number of different outputs you can choose from, and are listed here:

  • Event Hub
  • SQL Database
  • Blob Storage/Data Lake Storage Gen2
  • Table Storage
  • Service Bus Topic
  • Service Bus Queue
  • Cosmos DB
  • Power BI
  • Data Lake Storage Gen1
  • Azure Function

Additional information is available for each of these types here.

streamanalytics7
Figure 7: Stream Analytics Outputs

For our example I will be configuring an output to Cosmos DB. Adding an output of this type displays a blade that allows entry of the details of the output being created. You need to enter an alias name for the output, along with the choice of entering the Cosmos DB setting manually or selecting from your Azure subscription. We are selecting an existing Cosmos DB resource that was created previously. Please note that you must have previously created a container in Cosmos DB before specifying the Container Name and Document Id values.

streamanalytics8
Figure 8: Adding CosmosDB Output

Queries

Once you have one or more inputs and outputs configured, you can now create a query to process the incoming data. Queries are the mechanism used by Stream Analytics to transform the incoming data in real time. These queries are written using Stream Analytics Query Language which are similar to SQL statements.

Structurally the query has a SELECT statement which selects data from one or more configured inputs and uses an INTO clause to specify an output to emit the transformed data to. There are also FROM, JOIN and WHERE clauses which help specify and filter the incoming data and perform processing against it.

In our example, when you click on the Query section under Job Topology, a page is displayed that shows the current query for this Stream Analytics job. You are able to edit in the window to specify your select query statements. Notice that we are using SELECT * wildcard to specify all fields from the input source and saving those to the specified output.

streamanalytics9
Figure 9: Stream Analytics Queries

You can see preview data displayed from our input source (IoT Hub device):

streamanalytics9-1
Figure 10: Input Preview Data

This querying editor also has the ability to perform discrete field selection and filtering. So, for example if we wanted to specify certain fields we could modify our query to look like this:

streamanalytics10
Figure 11: Discrete Field Selection in Queries

Additionally, if we wanted to filter the incoming data we have that ability as well. For example, if we wanted to exclude any incoming temperature data that did not exceed 100 we could specify a filter like this:

streamanalytics11
Figure 12: Query Filtering

Once we have the inputs, outputs and queries configured we are ready to turn on our Stream Analytics job. Looking at the Overview page we can see at a glance the configured inputs, outputs and what the query is specified as.

When we start the Stream Analytics job and wait a few minutes, we should begin to see data coming in and being processed. The Monitoring and Resource Utilization graphs would show some data coming through as it is being processed.

streamanalytics12
Figure 13: Stream Analytics Graphs

We can also see our data appearing in our CosmosDB database, as shown here:

streamanalytics13
Figure 14: CosmosDB Data Records

Conclusion

In conclusion, Azure Stream Analytics provides the ability to process complex events in real time, with the purpose of providing real time analytics to incoming data sources. We also showed an example on how to create and configure Azure Stream Analytics to accept incoming IoT device telemetry data and process and save it into a CosmosDB database. I hope this article has been helpful and please click here for more information.

Cheers!

Azure IoT Hub Messaging

The Azure IoT Hub service provides the ability for bi-directional communication with your devices. You can use Azure IoT Hub to send messages from your devices to your IoT cloud solutions, as well as send commands from your IoT cloud solutions to your devices. This article will describe the different types of communications possible with Azure IoT Hub and how you can use them.

Prerequisites

Before you can send/receive messages between Azure IoT Hub and your devices you need the following:

  • An Azure account. Sign up for a free trial here
  • Provision an instance of IoT Hub
  • Create at least one device with your IoT Hub

Bi-directional communication means just that – in both directions. What that means for Azure and IoT is that you either send data from your devices to IoT Hub, or data or commands from IoT Hub to your devices. I will describe the different options to accomplish this, as seen below:

  • Device-to-Cloud messages
  • Cloud-to-Device messages
  • Device Twins
  • File Upload
  • Direct Methods

Device-to-Cloud Messages

Sending information from your IoT devices to the cloud is supported by Azure IoT Hub with device-to-cloud (D2C) messages. Your devices can send telemetry time series and alert data to IoT Hub to be routed to other services for processing.

Azure IoT Hub comes with a built-in endpoint that back-end services can use to read telemetry data being sent from your devices. This endpoint is compatible with Event Hubs and you can also use standard IoT Hub SDKs to read from this endpoint.

When D2C messages are sent from your devices to IoT Hub, by default they are routed to this built-in service endpoint (messages/events). This endpoint is exposed to your devices with MQTT, AMQP and HTTPS protocols. You also have the ability to route these messages to custom endpoints that you create.

Messages can be routed to multiple endpoints, so if you create custom endpoints and setup the routing queries you can have a message be routed to more than one endpoint. IoT Hub currently supports the following custom endpoints:

  • Azure Storage (Blob or Data Lake Gen2)
  • Service Bus Queues
  • Service Bus Topics
  • Event Hubs

Here is an example route that filters out messages that have a temperature alert flag specified and routes the messages to the default events endpoint.

Figure 1: IoT Hub Routing Example

Please note that the built-in Event Hubs endpoint (messages/events) is the fallback route, so if incoming device messages do not satisfy query conditions for any custom endpoints they will be sent to this fallback route. You must enable the fallback route capability when you turn on message routing or your incoming messages will not be routed properly.

D2C messages received are stored temporarily by IoT Hub up to 7 days. They are lost if not processed by a configured back-end service. Messages can also have a maximum size of 256KB for each message. The throughput of what an IoT Hub can handle depends on the pricing tier. Details are here.

Additional details on D2C communications can be found here.

Cloud-to-Device Messages

Sending one-way communications from a cloud solution to one or more devices is possible with cloud-to-device (C2D) messages. Your cloud solutions can send notifications to devices, or commands that can alter the state of devices (device twins).

Azure IoT Hub will send messages to devices through a service-facing endpoint (messages/devicebound), and the device(s) then receive the messages through a device-specific endpoint (devices/{deviceid}/messages/devicebound).

Azure IoT Hub provides a guarantee that messages will be sent to devices at-least-once. In order to do this, IoT Hub persists C2D messages in per-device queues. These messages persist in the queues until the devices explicitly acknowledge completion (aka – they received the message successfully). Once acknowledged, IoT Hub removes the message from the appropriate per-device queue. This approach guarantees resiliency in the case of connectivity and/or device failures.

Additional details on C2D communications can be found here.

Device Twins

JSON documents that contain device state information are what is known as device twins. These documents store the metadata, configurations and conditions for a device. Azure IoT Hub maintains a device twin for each device you connect to IoT Hub. There are multiple uses that device twins provide:

  • Store device-specific metadata in the cloud
  • Report current state information about the device
  • Synchronize the state of long-running workflows between the device and cloud solution
  • Querying device metadata, configuration or state

The structure of a device twin is shown in Figure 2:

IoTHubMessaging2
Figure 2: Device Twins

Tags are a section of the JSON document that the cloud solution can read/write to. Devices cannot see these tags or their values.

Desired Properties are a section of the JSON document used to synchronize device configurations or conditions. The cloud solution can set these properties and the device can read them. The device can also receive notifications that the desired properties have changed.

Reported Properties are a section of the JSON document also used to synchronize device configurations or conditions. Devices can set these properties and the cloud solution(s) can read them.

Root object properties define the device identity and also provide container objects for tags and the reported, desired properties sections.

Here is an example of a device twin:

{
    "deviceId": "testDevice1",
    "etag": "AAJWKSOAAAAAc=", 
    "status": "enabled",
    "statusReason": "provisioned",
    "statusUpdateTime": "0001-01-01T00:00:00",
    "connectionState": "connected",
    "lastActivityTime": "2020-02-28T12:44:48.976Z",
    "cloudToDeviceMessageCount": 0, 
    "authenticationType": "sas",
    "x509Thumbprint": {     
        "primaryThumbprint": null, 
        "secondaryThumbprint": null 
    }, 
    "version": 1, 
    "tags": {
        "$etag": "123",
        "deploymentLocation": {
            "building": "A",
            "floor": "1"
        }
    },
    "properties": {
        "desired": {
            "telemetryConfig": {
                "sendFrequency": "5m"
            },
            "$metadata" : {...},
            "$version": 1
        },
        "reported": {
            "telemetryConfig": {
                "sendFrequency": "5m",
                "status": "success"
            },
            "batteryLevel": 55,
            "$metadata" : {...},
            "$version": 4
        }
    }
}

The cloud solution backend can operate on the device twin using these operations (through HTTPS):

  • Retrieve Device Twin: Returns the JSON document for the device twin
  • Update Device Twin: Enables the partial update of the JSON document. Tags and Desired Properties can only be updated. Cloud solution can also add or remove properties in these sections. Setting a property to null removes it from the section. There is also the ability to completely overwrite the Tags and Existing Properties with new JSON.
  • Receive Device Twin Notifications: Allows the cloud solution to be notified when a change is made to the device twin. A route in the IoT cloud solution called twinChangeEvents must be created and configured as a route for this operation to succeed.

The device can operate on its device twin using these operations:

  • Retrieve Device Twin: Returns the JSON document for the device twin. Tags are not visible.
  • Update Device Twin: Enables the partial update of the JSON document. Reported Properties can only be updated. This operation works in the same way as the cloud solution updates a device twin.
  • Observe Desired Properties: The device can choose to be notified of updates to the Desired Properties when they occur.

There are size limitations on the different sections in the device twin, as seen here:

  • Tags: 8KB max size on values
  • Desired Properties: 32KB max size on each value
  • Reported Properties: 32KB max size on each value

As an example, if the cloud solution wants to change the sendFrequency value for the device it modifies the device twin value for sendFrequency, as seen here:

"desired": {
   "telemetryConfig": {
   "sendFrequency": "5m"
},
…

The device(s) would be notified of this change immediately or when they become connected (if offline). The device would then report back to the cloud solution the updated configuration on the device:

"reported": {
   "telemetryConfig": {
   "sendFrequency": "5m",
   "status": "success"
},
…

File Upload

IoT devices can start file uploads through a device-facing endpoint in IoT Hub. You can have your devices request file uploads to an associated Azure storage account in the cloud, with IoT Hub acting as a dispatcher for the uploaded file (requested by a device) to the Azure storage account. File uploads can be used for large files such as media files or large telemetry batches.

In order for file uploads to work, you must first associate an Azure storage account to IoT Hub. This can be done via the Azure Portal or programmatically through the IoT Hub REST APIs.

IoTHubMessaging3
Figure 3: File Upload Section

Figure 4: File Upload Configuration in IoT Hub

Once you have made this association you can write code in your device to upload files to the cloud. IoT Hub will handle the upload to its associated blob storage account and provide notifications that you can monitor. Here is a great tutorial that reviews the steps needed to perform a file upload from your device.

Direct Methods

Direct methods provide an alternative way of communication with devices that work independently of the other communication options. Direct methods allow us to call a method on the device as a local service, which means we can get immediate feedback from the results of that call.

Direct methods are implemented on the device and use a request-response pattern when communicating. You can invoke a direct method through a service-facing URI and are a synchronous call. The invocation will either succeed or fail (after the timeout period), returning a result with an optional payload. The benefits/drawbacks of direct methods are as follows:

Benefits

  • Define multiple named direct methods
  • Pass an optional payload
    Returns a status code and optional payload
  • High throughput

Drawbacks

  • Only supports AMQP or MQTT
  • Doesn’t work if device is offline

As an example, here is how you would write code on your device to receive calls to a direct method:

await device.SetMethodHandlerAsync(“showMessage”, ShowMessage, null);
…


private static Task ShowMessage(MethodRequest methodRequest, object userContext)
{
…
}

Calling the SetMethodHandlerAsync class registers a named method handler with the device, which in this example the method name is ShowMessage. Parameters are the name of the message handler method as well as the reference to that handler method and an optional context object.
The definition of the ShowMessage handler method is also defined, which takes a MethodRequest object parameter and optional context object and returns a Task object.

Now that our devices are able to handle direct messages from the cloud, the only other thing to do is write code in our IoT cloud solution to call the direct method on our device. We can do this by defining a method to invoke the direct method call on our device, as seen in this console application example:

static async Task Main(string[] args)
{
   …
   var serviceClient = ServiceClient.CreateFromConnectionString(svcConnString);
   await CallDirectMethod(serviceClient, deviceId);
   …
}


private static async Task CallDirectMethod(ServiceClient sClient, string deviceId)
{
   var method = new CloudToDeviceMethod(“showMessage”);
   method.SetPayloadJson(“’This is a test direct method’”);

   var response = await sClient.InvokeDeviceMethodAsync(deviceId, method);

   Console.WriteLine($”Response status: {response.Status}, payload:{response.GetPayloadAsJson()}”);
}

As you can see, the Main method creates an instance to our IoT hub using the ServiceClient class. We then create a private method named CallDirectMethod(), which takes the instance of our ServiceClient class along with the Device ID value. This method instantiates the direct method via the CloudToDevice() class, which we specify the name of the direct message when instantiating (showMessage in our example). We can then set the payload JSON and asynchronously invoke the direct method on the device. A response is returned to let us know if the method ran successfully or not.

Conclusion

In conclusion, we described the different types of communications possible with Azure IoT Hub and how you can use them. I hope this article has helped you understand the different types of Azure IoT Hub messaging.

Cheers!

Azure IoT Central

Azure IoT Central is Microsoft’s SaaS (software-as-a-service) offering that allows you to quickly provision an IoT solution. This service reduces the burden and cost of developing, managing and maintaining enterprise-level IoT solutions, allowing you to focus more on your IoT data and how it can transform your business. I will be going through the steps to create an Azure IoT Central application and how easy it is to create and maintain an IoT solution.

You can use IoT Central to create a custom, cloud-based IoT solution without having to worry about the details of the underlying infrastructure. Customization of your solution can all be done from the browser in the application UI, and includes capabilities to:

  • Define the layout of properties and settings on a device template
  • Configure custom dashboards
  • Configure custom analytics to explore device data

 

Prerequisites

The only prerequisites needed are a Microsoft account (personal, work or school) as the creation process will ask you to login to your account, and an Azure subscription.

 

Getting Started

First you need to navigate to the Azure IoT Central Build site. You will notice a left-hand navigation menu which consists of several sections:

IoTCentral1

Figure 1: Azure IoT Central Menu

 

Clicking on the Build menu selection will take you to a page that displays a number of options to build your IoT application. You can start with a number of pre-generated templates that are geared towards specific industries or start with a blank application template.

Once you select either an industry-specific or custom template, a page displays that allows you to specify information about your application.

IoTCentral2

Figure 2: Creating an IoT Central Solution

The Application Name is pre-populated for you, as well as the URL. Please note that the application name and URL must be globally unique. The Application Template dropdown is only visible on custom applications, as you’re already selecting a specific template if you choose one of the industry-specific templates.

You also need to choose the pricing plan – you can try the solution for free for 7 days or select one of the other pricing tiers. These other tiers are based on how often you want to send messages from your devices. The Billing Info allows you to select the Directory (Azure Active Directory) and Azure Subscription you wish to have this solution hosted from. Location is also required as the system will provision the IoT Central solution resources in the closest region.

 

Web Application

Once the system provisions your IoT Central application you are taken to the Dashboard page, which displays some starting links such as adding devices, quick start demos, tutorials and documentation.

IoTCentral3

Figure 3: Dashboard page

 

Devices

Adding devices is easy in IoT Central – just click on the Devices tab and you are taken to a page that displays all the devices currently in the application. Adding a device is done by clicking on the +New button. A dialog window is displayed allowing you to configure the details about the new device being added:

IoTCentral4

Figure 4: Adding a new device

Device ID is a unique identifier that is used by the device to connect, while Device Name is a friendly name for the device. You can also toggle the Simulated option which specifies whether your device is simulated or real.

 

Device Templates

You can also create one or more device templates, which can then be used to create new devices that have the template characteristics. Click on the +New button on the Device Templates tab (under App Settings) and you can select the template type:

IoTCentral5

Figure 5: Device Templates

You have the ability to create a template for an IoT device, Azure IoT Edge device, or one of the pre-configured device templates listed. The list of certified devices (with IoT Plug and Play) are shown here and more are being added all the time.

Once you publish your device template, you can add devices based on that template. On the Devices tab you will notice a selection with the name of your device template that you can click on. This will display a list of the current devices that are based on this device template. Clicking the +New button will allow you to create a device based on this template.

IoTCentral6

Figure 6: Creating a Device using Device Template

 

Device Groups

Device Groups are a logical grouping of devices in the system based on specific attributes. You can create one or more Device Groups to filter one or more devices in your system. This is useful if you have hundreds (or thousands) of devices and you need to look at specific devices quickly.

Navigate to the Device Groups tab and click on the +New button to create a new device group. You will see a page that allows you to enter information for your new device group such as the Name and Scope. Adding one or more filters in the Scope area will define the filters needed to only display the device(s) you want to see.

IoTCentral7

Figure 7: Device Groups

 

Rules

You can create telemetry-based rules in IoT Central. This is done under the Rules section in the application. Clicking on the Rules tab will display a list of the current rules in the system. Create a new rule by clicking on the +New button. This will display a page that allows you to create a new rule.

IoTCentral8

Figure 8: Rules

Give the rule a name and then specify a target device template, which will define the devices to which the rule applies. If you need to specify less than all the devices in the device template you select, create one or more filters to filter the list of devices further.

Conditions are created to specify when the rule is triggered. You can specify one or more conditions the rule must meet to be triggered.

There is also a section for Actions. These are the things that happen once the rule has been triggered. You notice you can either send an email or call a web hook when the rule is triggered. You can create one or more actions if you choose.

IoTCentral9

Figure 9: Email Action

 

IoTCentral10

Figure 10: Webhook Action

 

Analytics

Azure IoT Central also has the capability to analyze device data including time series information. You can specify the telemetry values to display along with splitting the data collections by attributes (such as Device ID). Select the Device Group and Telemetry value(s) to analyze and click the Analyze button when you are finished.

IoTCentral11

Figure 11: Analytics

 

Jobs

You can use Azure IoT Central to manage your connected devices using jobs. Jobs are useful for performing bulk updates to your device properties or for running commands. Clicking on the Jobs tab will display a list of the existing jobs in the system, job status and when they were last run. Creating a new job will allow you to specify details of the job such as Name, what device group the job targets as well as the Job Type and detailed information.

IoTCentral12

Figure 12: Creating a Job

Running a job is done by clicking on the Run button. This will execute the job and return results of the operation.

IoTCentral13

Figure 13: Running a Job

 

Administration

The IoT Central application also allows administration of the website, including Users in the system, Roles, Pricing and the Device Connections. You also have the ability to permanently delete the solution.

IoTCentral14

Figure 14: Administration

 

Underlying Infrastructure

Since IoT Central is a SaaS (software-as-a-solution) offering, you do not have access to the underlying architectural components in Azure. What this means is that Microsoft fully hosts and supports the IoT Central application but they also manage the resources themselves. You do not have the ability to configure/customize/remove any of the individual Azure resources used in your IoT Central application. If you need to customize the IoT Central application beyond what is provided to you through the SaaS offering, then you need to consider Azure IoT Solution Accelerators which do provide the ability to manage and modify the underlying Azure components.

 

In conclusion, Azure IoT Central is a nice offering from Microsoft if you want to get your IoT solutions up and running quickly! Device provisioning and customizations are part of this service, as well as Rules, Jobs, and analytics for the device data. I hope this brief tour of Azure IoT Central has been helpful to you.

Cheers!

Message Routing with Azure IoT Hub

IoT solutions can be complex, especially if you have multiple devices that you need to send to more than one destination in your cloud solution. Azure IoT Hub provides the capability to send your device data to multiple destinations through message routing. This article will describe the available options with message routing and how they are used.

IoT solutions can be complex, especially if you have multiple devices that you need to send to more than one destination in your cloud solution. Azure IoT Hub provides the capability to send your device data to multiple destinations through message routing. This article will describe the available options with message routing and how they are used.

Prerequisites

Before you can use message routing with your devices you will need the following:

  • An Azure account. Sign up for a free trial here
  • Provision an instance of IoT Hub
  • Create at least one device with your IoT Hub

Incoming Messages

The devices in your IoT solution will be sending telemetry data into Azure IoT Hub. Conceptually, you normally think of these messages all being sent to one place and being handled/processed by the hub. However, what if you want these messages sent to different cloud services that may be part of your solution? In the past you might have had custom code to handle these messages and route them appropriately. That made your solution overly complex when you were required to handle all the details about these incoming messages, such as handling transient faults, how to deal with lost messages, and maintaining high reliability.
Fortunately, Microsoft has made this easier for us by providing message routing as part of Azure IoT Hub. IoT Hub will now take care of all those complex details for routing messages so we can focus on our custom IoT solution code. IoT Hub provides scalability for incoming messages, handling transient faults, and how to handle lost messages. The details of message routing are abstracted from us, and we only need to know how to configure the routing to specify where we want our messages to go.
MessageRouting1

Figure 1: Message Routing

 

Endpoints

There are a number of places you can route your device messages to by configuring IoT Hub. These are the final destinations of your data, allowing backend processing services to execute actions against the data you route to it. The different types of endpoints are as follows:

  • Event Hub
  • Service Bus Queue
  • Service Bus Topic
  • Storage

Also note that paid SKU hubs can have up to 10 additional endpoints while free SKU hubs can only have 1 additional endpoint.

Event Hub

Creating an event hub allows you to route your device data to a specific event hub resource to allow further/dedicated processing. Note that IoT Hub comes with a built-in endpoint that is compatible with this type. Here is an example of adding a new event hub endpoint to IoT Hub:

MessageRouting2

Figure 2: Event Hub Endpoint

 

Service Bus Queue/Topic

You can create a custom endpoint to a Service Bus Queue/Topic, which would then create a record that can be processed by an Azure function for example. Here is an example of adding a new Service Bus Queue endpoint to IoT Hub:

MessageRouting3

Figure 3: Service Bus Endpoint

Creating a custom endpoint to a Service Bus Topic works in the same way you would create one for a Service Bus Queue, but please note that your Service Bus Namespace must be the Standard (or higher) pricing tier for topics to be available. Also be aware that you cannot create a custom endpoint to a Service Bus Queue/Topic if they have sessions or duplicate detection enabled.

Storage

You can also create a custom endpoint that uses Azure blob storage, allowing the saving of incoming messages to an existing container (as blobs only). Here is an example of adding a new Azure Storage endpoint to IoT Hub:

MessageRouting4

Figure 4: Storage Endpoint

Batch Frequency specifies (in seconds) the maximum amount of time before the system will batch and write records to Azure Storage. The Chunk Size Window specifies the maximum size of blobs (in MB). Apache AVRO is the default encoding, although you can specify JSON.

Routes

Routes are created/configured to direct the appropriate data source to an endpoint for processing. When you create an endpoint you must specify the Name, Endpoint (built-in or custom), Data Source, and filter query.

MessageRouting5

Figure 5: Adding a Route

The Data Source field specifies where the data is coming from and must be one of the following selections:

MessageRouting6

 

You also have the ability to filter the incoming message data, which can be done using queries in the Routes. IoT Hub defines a common format for all device-to-cloud messaging, and filtering is done by writing a query against this common format. Here is an example:

{ 
  "message": { 
    "systemProperties": { 
      "contentType": "application/json", 
      "contentEncoding": "UTF-8", 
      "iothub-message-source": "deviceMessages", 
      "iothub-enqueuedtime": "2017-05-08T18:55:31.8514657Z" 
    }, 
    "appProperties": { 
      "processingPath": "{cold | warm | hot}", 
      "verbose": "{true, false}", 
      "severity": 1-5, 
      "testDevice": "{true | false}" 
    }, 
    "body": "{\"Weather\":{\"Temperature\":50}}" 
  } 
}

Querying on message system properties should prefix a ‘$’ symbol on the query. Queries on application properties do not need this symbol prefixed. For example, querying on the system property ‘contentEncoding’ would look like this:

$contentEncoding = 'UTF-8'

Querying on an application property would look like this:

processingPath = 'hot'

Please note that device-to-cloud messaging is not case-sensitive, so using unique property names is recommended. Here is an example of a query that filters records that have the application property ‘temperatureAlert’ = true.

MessageRouting7

Figure 6: Message Routing Queries

A fallback route is present so that any messages that do not satisfy any specified query conditions will be sent to this route. The default fallback route is the ‘messages/events’ route that is used by the built-in endpoint (compatible with Event Hub). You can turn on message routing in IoT Hub, which will allow you to create custom routes. Once a custom route is created, data stops flowing to the built-in endpoint unless you specifically create a route to that endpoint. Please be aware that if you delete all custom routes and you disable the fallback route, data will not be sent to any destination.

 

In addition to telemetry data, routes can be created for other events such as Device Twin changes, Device Lifecycle events and Digital Twin changes (currently in public preview). If you create a route with the data source set to one of these other events, IoT Hub will send messages to the appropriate endpoint for that event (Device Twin, Digital Twin, Device Lifecycle).

 

Conclusion

In conclusion, using message routing with Azure IoT Hub increases the flexibility, reliability and scalability of your IoT solution. Routing also enables you to keep track of specific events in your solution, such as device twins and device lifecycle. I hope this article has helped you understand the different options for message routing with Azure IoT Hub.

Cheers!

Creating and Configuring Azure IoT Hub

Microsoft provides the Azure IoT Hub service to assist in implementing Internet of Things (IoT) solutions. This fully managed Azure service enables reliable and secure bidirectional communication between your IoT devices and a back-end solution hosted in Azure. I am going to review the steps to create an IoT hub in Azure, along with how to register a device and configure tier and scaling options.

Prerequisites

The only prerequisite you need to follow along is an Azure account. You can sign up for a free trial here.

Creating an IoT Hub Instance

Azure Portal

Probably the easiest way to create an instance of IoT Hub is to use the Azure portal. You do this by first logging into the Azure portal with your Azure credentials, and then clicking on Create a Resource in the upper left. This displays a new blade with a number of choices under Azure Marketplace. Click on Internet of Things and you should see the IoT Hub selection, as seen here:

CfgIoTHub2

Clicking on IoT Hub will display a new blade that allows you to enter in details of your new IoT Hub, such as the Subscription, Resource Group, Region and IoT Hub name. The Subscription is auto filled with the subscription you are currently logged into. You can select a Resource Group to place the new IoT Hub in or create a new one. Region specifies the location that your IoT Hub will be created in, and the IoT Hub Name is a globally unique name you give to your IoT Hub.

CfgIoTHub3

Clicking Next: Size and Scale takes you to the next screen which allows you to select a pricing tier for your IoT Hub, along with the number of IoT Hub units allocated. The pricing tiers include a Free Tier as well as 3 levels of either Basic or Standard Tiers (B1-3, S1-3 respectively). The choice you make here determine a number of options relating mainly to data throughput and scaling. Details are here.

At the time of this writing, the highest pricing tier selection is S3: Standard Tier as shown below:

CfgIoTHub4

Clicking Review + create will take you to a summary page listing out the selection choices you made. Clicking Create here will instruct the Azure Portal to create the IoT Hub with the choices you selected.

CfgIoTHub5

Azure CLI

Another way to create an instance of IoT Hub is by using the command-line interface (CLI). Azure hosts an interactive shell environment called Azure Cloud Shell that you can use to execute CLI commands. You can launch Azure Cloud Shell by either going to https://shell.azure.com in your browser, or by clicking on the Cloud Shell button in the upper right in the Azure Portal.

CfgIoTHub9

Once you are in your Cloud Shell environment, you can execute the following command to create an instance of IoT Hub:

az iot hub create --name {your iot hub name} --resource-group {your resource group name} --sku S1

The only required parameters are the name of your IoT Hub along with the resource group it will be associated with. Insert a globally unique name for your IoT Hub along with the name of an existing resource group. The optional –sku parameter designates what pricing tier to create the IoT Hub with. Pricing details are located here.

There are additional optional parameters when creating an IoT Hub and are listed here.

Configuring IoT Hub Tiers

There are two tiers for Azure IoT Hub: Basic and Standard. You should consider what your IoT solution is designed to do before you select a tier. The difference between Basic and Standard tiers are the number of features they support. A Free tier is also provided, which has all the functionality of the Standard tier but has limited messaging allowances.

If your solution collects data and you are analyzing it centrally, then the Basic tier is probably the best choice. If you want to use advanced configurations to control your IoT devices remotely, or you want to distribute some of your workloads to the devices themselves, then the Standard tier is probably the best choice.

Each tier has three levels, listed in numerical order (1, 2, 3). The difference between these three levels is how much data they allow to be send from devices per day:

  • B1, S1 tiers: 400,000 messages/day/unit
  • B2, S2 tiers: 6 million messages/day/unit
  • B3/S3 tiers: 300 million messages/day/unit

The following features are available in all IoT Hub tiers:

  • Device-to-cloud telemetry
  • Per-device identity
  • Message routing, enrichments and Event Grid integration
  • HTTP, AMQP, and MQTT protocols
  • Device Provisioning Service
  • Monitoring and diagnostics

The following features are only available in the Free/Standard IoT Hub tier:

  • Cloud-to-device messaging
  • Device twins, Module twins, and Device Management
  • Device streams (preview at the time of this blog)
  • Azure IoT Edge
  • IoT Plug and Play (preview at the time of this blog)

Registering Devices

Once you have your IoT Hub instance created, you can register one or more devices identities that you intend to use with your IoT Hub.

Azure Portal

After you log in to the Azure Portal, navigate to your IoT Hub and click on the IoT Devices selection (under Explorers section):

CfgIoTHub6

This will display a list of the current devices created for the IoT Hub. You can click on the + New link towards the top of the page and a new blade will be displayed, allowing you to enter the following information about your device:

  • Device ID – enter a name for the device. This is used for authentication and access control
  • Authentication Type – select how the device will authenticate to IoT Hub. Symmetric Key is the default and allows entry of the primary/secondary keys (see below). X.509 Self-Signed and X.509 CA Signed allow you to select security certificates for the device (certificate created prior)
  • Primary/Secondary Keys – allows you to set a unique value for accessing the device. Disabled if auto-generate keys is checked (valid for Symmetric Key authentication type only)
  • Auto-generate keys – this will instruct Azure to generate unique values automatically for the primary/secondary keys (valid for Symmetric Key authentication type only)
  • Connect this device to an IoT Hub – specifies if this device will interact with this IoT Hub
  • Parent Device – specify if this device has a parent device. Used for IoT Edge

CfgIoTHub7

Clicking Save will instruct the Azure Portal to create the device with your selections. Once completed, you can access the device details by clicking on its name in the list of devices:

CfgIoTHub8

Azure CLI

You can also register device identities with your IoT Hub using Azure CLI. Execute the following command in your Cloud Shell instance, replacing {iothub_name} with your IoT Hub name and {device_id} with the name you wish to use for your device:

az iot hub device-identity create -n {iothub_name} -d {device_id}

The console will emit a JSON string upon success which give information about the device you just registered, such as primary/secondary key values, its connection state and whether it has IoT Edge capabilities:

CfgIoTHub10

You can verify in the Azure portal that the device is registered by navigating to the IoT Devices section in your IoT Hub. Your newly-registered device should be displayed in the list:

CfgIoTHub11

Additional parameters can be found here.

I hope this article has helped you understand what Azure IoT Hub is and how to create an instance and register your IoT devices to it, so you can begin sending data to Azure.

Cheers!

Using IoT on a Beer Kegerator

Being born and raised in the great state of Wisconsin, beer has been a part of most of my adult life. Couple that with my love of technology, I always wondered how I could leverage some cool tech with a beer theme. Since the proliferation of inexpensive hardware and the Internet of Things (IoT), it has now become easy (and cheap!) to provide solutions that can be used to monitor (among other things) beer-related activities. This article will describe and detail the steps I took to create a solution for monitoring beer consumption on a beer kegerator.

The first thing I needed to do before building anything is to understand and design what it is I wanted to build. Since I want to monitor beer consumption from a kegerator, I needed to draw out the major parts of my solution. Once I know that I can then begin to build and test the different parts of the system. The drawing below shows the major parts of my solution:

1

As you can see, when someone taps a beer from the kegerator, an inline flow meter sensor sends information to an IoT device, which then processes the information and sends it to the cloud, where it is stored for data analysis.

Now that I have an idea of my overall architecture, I can begin to think about what hardware and software I need to create my solution.

 

Hardware

For hardware, I chose to use a Raspberry Pi as my IoT device. The Pi is a low-power, inexpensive device that met my needs for this project (built-in ethernet network, multiple GPIO pins, easy to install apps). Please note that I also considered using the ESP8266 chip for this project – this little chip is great for simple IoT project as it’s really cheap, has built-in wireless networking (with a full TCP/IP stack!), and multiple GPIO pins for use. The main drawback for this project is that this chip only provides 3.3v for power and I needed 5v for the flow sensor, so it was easier to use the Pi. The other drawback is that I can’t install Windows 10 IoT Core on the ESP8266, so using a Pi simplified my design.

The other piece of hardware I need is a flow sensor to measure the flow of beer through the line when it’s being tapped. Initially I chose a really cheap sensor designed for coffee-makers but found out that these won’t work for measuring beer flow (see Testing section), so I went with a more expensive sensor. I chose the Swissflow SF-800 (link), which is about $60 USD. This flow sensor sends digital pulses when a liquid is flowing through it, so that allows me to measure how much beer is being dispensed. This sensor requires +5Vdc to power it properly, so that required me to use a Raspberry Pi (which also provides +5Vdc).

 

Software

The software selections I made were driven (in part) by my hardware choices, but also by what apps I wanted to provide. I wanted to have an app that runs on the Raspberry Pi and processes the incoming pulse data from the SF-800 sensor and then send that data to Azure. I also wanted this app to have a user interface that displayed how much beer is left in the current keg, along with the ability for an administrator to “reset” the app (when the keg is empty and is changed out for a full one).

Windows 10 IoT Core provides the operating system for the Raspberry Pi, and this also allows me to easily deploy and manage any apps I want running on the device. Please review this link on how to install Windows 10 IoT Core on the Raspberry Pi.

2

The app that I am creating for this solution is a Universal Windows Platform (UWP) app and is designed for running on IoT devices that have Windows 10 IoT Core on them. This app will process the incoming digital pulses from the SF-800 and send them to Azure IoT Hub.

3

The following code snippet shows how I receive the incoming digital pulses from the SF-800 flow sensor. I have this sensor connected to GPIO pin 5 from the Raspberry Pi so that when the value on that pin changes it triggers an event in my app to signal that a pulse was sent by the SF-800.

4

I also have a timer on another thread that ticks every 0.5 second and looks to see if any incoming pulses have been received by the SF-800 flow sensor. If there have, it sends them off to Azure IoT Hub for storage.

5

The software in the Azure cloud that I will be leveraging is Azure IoT Hub, Stream Analytics and Azure SQL. Azure IoT Hub provides the mechanism to receive incoming telemetry data from my IoT device and route it for processing and storage. I am having Azure IoT Hub route my data to Stream Analytics, which then will process it and save it in an Azure SQL database. Once in the database, I am free to consume it in a number of ways, such as PowerBI or any custom app that can consume data from SQL.

6

As incoming telemetry data is received from the Raspberry Pi, Azure IoT Hub receives that data and Stream Analytics is used to process that incoming data and save it in an Azure SQL database. This is done through the Stream Analytics interface by setting up and input (Azure IoT Hub) and an output (Azure SQL database) and configuring a query to do any processing needed at that time.

7

Testing

Once I created the software components and connected the hardware that I have, it is time to test the functionality of my solution. I first tested the solution by connecting my Raspberry Pi (with my UWP app installed) to a breadboard where I have the SF-800 flow sensor connected. I also have a couple of LEDs to indicate a heartbeat pulse (green) and to indicate flow sensor pulses (red).

8

I configured Azure IoT Hub and started my Stream Analytics job so that incoming data from my IoT device will be received and processed properly. Testing this way involved blowing air through the SF-800 device (I used my breath – GENTLY!), making sure the air flow was in the proper direction (going the wrong way can damage the sensor).

Once I knew this was working I wanted to validate the accuracy of the digital pulses of the SF-800. To do this, I got some plastic tubing of the same size being used in the kegerator along with a funnel. I then measured out 1 cup of water and then proceeded to pour it through the flow sensor while everything was running.

9

Deployment

Now that I have tested my solution, it is ready for deployment! This included placing the flow sensor inline with the actual kegerator tubing on the line I wanted to monitor. I still kept the breadboard as this was not a fully productized solution (meaning I didn’t create the wiring on a PCB).

10

I encountered a testing issue I failed to realize until after I deployed my solution for the first time. I was originally using a cheap flow sensor designed for coffee makers, and when I deployed this to the beer line I noticed that it made the beer foam as it was passing through the sensor. This was something I didn’t test for prior to deployment so it forced me to rethink my design (and what sensor to use). I eventually found the SF-800 sensor and this worked much better when I deployed it with my solution.

In conclusion, now that this solution is connected to the kegerator, I can monitor how much beer is left in the current keg! I can also enhance my solution by leveraging an Azure Webjob to send an email notification when the keg is getting low. How great is that? No more tapping a beer just to find out that there isn’t any left!

Cheers!

11