Part 1 - Gathering pollution data with RaspberryPi, pushing them to AWS IoT Core and DynamoDB (TCH)

Fabio Beoni - Sep 2 - - Dev Community

This article is part of the series "Generative-AI-driven Chatbot for the Factory Plant" implementing a P.O.C. about an environmental pollution monitoring (PM2.5/PM10) to safeguard workers in a production plant.

This a techie article, if you are interested only on the high level use case aspects you can skip it.

Disclaimer

The information provided in this document is for informational purposes only and is subject to change. While efforts have been made to ensure the accuracy and reliability of the content, no guarantees are made regarding the completeness, reliability, or suitability of the information.
The author shall not be liable for any losses, injuries, or damages arising from the use or reliance on the information contained herein.

The AWS resources provisioned following this article can generate costs. Make sure to delete all the resources created at the end of the exercise.

Overview

In this section you focus on the IoT part (red box of diagram) configuring the gateway to:

  1. directly read data from the PM sensor
  2. authenticate to AWS IoT Core
  3. periodically push PM values to AWS IoT Core
  4. verify availability of the data

aws-diagram-iot-focus

1. Setup certificates and IoT cloud resources

The following steps assume you have:

  1. AWS CLI installed and configured
  2. OpenSSL installed
  3. NodeJS (v18 or v20) installed
  4. Optional - Terraform (>= v.1.9.5) if you prefer to work with IaC approach instead of running AWS CLI commands

1.1 Prepare the working location

Open the Terminal to start with the following commands:

# set required env variables for AWS CLI
export AWS_DEFAULT_REGION=<your-region>
export AWS_ACCOUNT_ID=<your-account-id>

# move to user home directory
cd ~/

# copy source code of the project to you directory
git clone git@bitbucket.org:fabiobeoni/bedrock-studio-chatbot.git
Enter fullscreen mode Exit fullscreen mode

Note: If you don't use Git, download the code from the same web address, and unpack it:



# move into the project directory
cd bedrock-studio-chatbot/pm-monitor-service

# install nodejs dependencies
npm install --production
Enter fullscreen mode Exit fullscreen mode

1.2 Create authentication certificate & key

Make the gateway device controller to connect to AWS IoT Core by authenticating it with certificate and private key.
You also want to associate the certificate to a virtual representation of the gateway known as IoT Things.

Give a name to the gateway device:

# set a variable to host the name
thing_name="gateway_pm_monitor"

# create certificate and key pairs, 
# assign the $thing_name as subject 
openssl req -x509  -newkey rsa:2048 -sha256  -days  365  -nodes \
    -keyout gateway.private_key.pem \
    -out gateway.certificate.pem \
    -subj "/CN=$thing_name"
Enter fullscreen mode Exit fullscreen mode

Download the Amazon Root CA1 certificate:

curl -O https://www.amazontrust.com/repository/AmazonRootCA1.pem
Enter fullscreen mode Exit fullscreen mode

Perform a ls -l in the local folder, to make sure it includes the following files:

AmazonRootCA1.pem
gateway.certificate.pem
gateway.private_key.pem
Enter fullscreen mode Exit fullscreen mode

1.3 Provision IoT resources

From now on, you can go ahead in two ways:

  1. provisioning the cloud resources by running AWS CLI commands (easy to follow step-by-step, requires manual deletion of the resources when done), or
  2. provisioning by IaC approach using the Terraform template (much faster and preconfigured, requires basic knowledge of Terraform - File included in the repo you just downloaded)

The article follows the provisioning by the AWS CLI, to make it easier to understand the process step-by-step.

From the terminal, provision a IoT Thing:

aws iot create-thing --thing-name $thing_name
Enter fullscreen mode Exit fullscreen mode

Then register the certificate into AWS IoT Core, to enable the device authentication:

# register certificate
certificatedata=$(aws iot register-certificate-without-ca \
    --certificate-pem file://gateway.certificate.pem \
    --status ACTIVE)

# keep track of the certificate Arn 
# and certificate ID in variables
certificate_arn=$(echo $certificatedata | jq -r '.certificateArn')  
certificate_id=$(echo $certificatedata | jq -r '.certificateId')
Enter fullscreen mode Exit fullscreen mode

At this point you can find the certificate in the AWS Web Console to:

# open the following address with a browser, make sure to replace
# <ASW_DEFAULT_REGION> with your region
https://<ASW_DEFAULT_REGION>.console.aws.amazon.com/iot/home/#/certificatehub
Enter fullscreen mode Exit fullscreen mode

Get the AWS IoT Core endpoint URL in your region, to push data from device to it:

# get the iot endpoint and save on a variable
endpoint_address=$(aws iot describe-endpoint --endpoint-type iot:Data-ATS --query  'endpointAddress' --output text)
Enter fullscreen mode Exit fullscreen mode

Create a client ID, a unique identifier for the gateway client:

# create a client ID
client_id=$(uuidgen)
Enter fullscreen mode Exit fullscreen mode

As always, in AWS anything must be authorized to perform a given task. The gateway device must connect and publish message data, so you want to provision a Policy to request the following permissions:

  1. iot:Connect
  2. iot:Publish 
# define the name and the "topic" you push data to 
policy_name="${thing_name}_iot_policy"
monitor_topic="pm_monitor"
topic_arn="arn:aws:iot:$AWS_DEFAULT_REGION:$AWS_ACCOUNT_ID:*"

# actually create the policy with required permissions
aws iot create-policy \
    --policy-name "$policy_name" \
    --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {"Effect": "Allow", "Action": ["iot:Connect"], "Resource": ["*"]},
    {"Effect": "Allow", "Action": ["iot:Publish"], "Resource": ["'"$topic_arn"'"]}
  ]
}'
Enter fullscreen mode Exit fullscreen mode

Now you attach the policy to the certificate, so that the gateway device authenticating with the certificate inherits the permissions listed in the policy, then attach the certificate to the thing:

aws iot attach-policy \
    --policy-name $policy_name \
    --target $certificate_arn

aws iot attach-thing-principal \
    --principal $certificate_arn \
    --thing-name $thing_name
Enter fullscreen mode Exit fullscreen mode

1.4 Provision database resources

Time for storing data. Right now the messages don't get persisted in any database yet. To save all incoming data from the gateway device you create a DynamoDB table (nosql database), and an IoT Rule (forwarding rule) to forward all incoming device data to it.

The items stored in the table have this JSON shape:

{
    // name of the company's plant where the gateway device and sensor reside and measure
    // (dynamodb partition key)
    "plant_name": "plant-abc", 

    // timestamp of the measure, when it happened
    // (dynamodb sort key)
    "date_time": 1633036800,

    // PM 2.5 value
    "pm25": 12.5,

    // PM 10 value
    "pm10": 20.3,

    //metadata about the machine generating the PM
    "machine": {
        "name" : "machine-one",
        "type" : "type-1",
        "code" : "mot1"
    }
}
Enter fullscreen mode Exit fullscreen mode

Create the DynamoDB table with:

dbtable_name=pm_values

aws dynamodb create-table \
    --table-name $dbtable_name \
    --attribute-definitions \
        AttributeName=plant_name,AttributeType=S \
        AttributeName=date_time,AttributeType=N \
    --key-schema \
        AttributeName=plant_name,KeyType=HASH \
        AttributeName=date_time,KeyType=RANGE \
    --provisioned-throughput \
        ReadCapacityUnits=5,WriteCapacityUnits=5
Enter fullscreen mode Exit fullscreen mode

You may want to feed the table with some synthetic data to test it out, before getting data from the gateway and IoT Core. If so, use the following script:

# current timestamp
current_time=$(date +%s)

# insert 20 items in a time range with different pm2.5 and pm10 values, they all refer to "plant-abc"
for i in {0..19}; do
    # Calculate the date_time for each item (1 hour apart)
    date_time=$((current_time - (i * 3600)))  # 3600 seconds in an hour
    pm25=$((RANDOM % 61 + 10))  # Random value between 10 and 70
    pm10=$((RANDOM % 61 + 10))  # Random value between 10 and 70

    aws dynamodb put-item \
        --table-name $dbtable_name \
        --item '{
            "plant_name": {"S": "plant-abc"},
            "date_time": {"N": "'"$date_time"'"},
            "pm25": {"N": "'"$pm25"'"},
            "pm10": {"N": "'"$pm10"'"},
            "machine": {
                "M": {
                    "name": {"S": "machine-one"},
                    "type": {"S": "type-1"},
                    "code": {"S": "mot1"}
                }
            }
        }'

done
Enter fullscreen mode Exit fullscreen mode

Run a query to read the data you just pushed:

# test the data inserted by querying dynamodb
aws dynamodb query \
    --table-name $dbtable_name \
    --key-condition-expression "plant_name = :plant_name" \
    --expression-attribute-values '{":plant_name": {"S": "plant-abc"}}'
Enter fullscreen mode Exit fullscreen mode

1.5 Provision the IoT Rule to store IoT data into the database

First, create role and policies to allow the data forwarding from IoT Core to the DynamoDB:

role_name=pm_monitor_iot_push_to_dynamo_role

# Trust policy – specifies the trusted accounts (iot.amazonaws.com) that are allowed to assume the role.
aws iam create-role \
    --role-name $role_name \
    --assume-role-policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "iot.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}'

# Permissions policy – grants the user of the role the required permissions to put items into the table.
policy_name=pm_monitor_iot_push_to_dynamo_policy

aws iam put-role-policy \
    --role-name $role_name \
    --policy-name $policy_name \
    --policy-document "{
        \"Version\": \"2012-10-17\",
        \"Statement\": [
            {
                \"Effect\": \"Allow\",
                \"Action\": \"dynamodb:PutItem\",
                \"Resource\": \"arn:aws:dynamodb:${AWS_DEFAULT_REGION}:${AWS_ACCOUNT_ID}:table/${dbtable_name}\"
            }
        ]
    }"
Enter fullscreen mode Exit fullscreen mode

Create an IoT Rule to direct messages from the topic to the table.

aws iot create-topic-rule \
    --region ${AWS_DEFAULT_REGION} \
    --rule-name "${monitor_topic}_to_dynamodb" \
    --topic-rule-payload "{
      \"sql\": \"SELECT * FROM '${monitor_topic}'\",
      \"description\": \"Rule to send messages from ${monitor_topic} topic to dynamodb ${dbtable_name} table\",
      \"actions\": [
          {
              \"dynamoDBv2\": {
                  \"putItem\": {  \"tableName\":  \"${dbtable_name}\"  },
                  \"roleArn\": \"arn:aws:iam::${AWS_ACCOUNT_ID}:role/${role_name}\"
              }
          }
      ],
      \"ruleDisabled\": false,
      \"awsIotSqlVersion\": \"2016-03-23\"
    }"
Enter fullscreen mode Exit fullscreen mode

1.6 Test the provisioned resources

To test the provisioned IoT resources, push a message to AWS IoT Core. To see messages coming from the test, subrscribe to the topic where the message is pushed to. This time you go ahead using the AWS Web Console.

First open the web browser to:

https://<AWS_DEFAULT_REGION>.console.aws.amazon.com/iot/home/#/test
Enter fullscreen mode Exit fullscreen mode

Second, in the web console create a subscription to the "pm_monitor" topic:


Now push a message to IoT Core:

# install the mqtt client
curl -L -o mqtt-cli-4.31.0.deb https://github.com/hivemq/mqtt-cli/releases/download/v4.31.0/mqtt-cli-4.31.0.deb
sudo apt install ./mqtt-cli-4.31.0.deb

# push a message to the topic "pm_monitor"
mqtt pub \
  -h $endpoint_address \
  -p 8883 \
  --cafile AmazonRootCA1.pem \
  --cert gateway.certificate.pem \
  --key gateway.private_key.pem \
  -t $monitor_topic \
  -m "{ \"plant_name\": \"plant-abc\", \"date_time\": 1633036800, \"pm25\": 12.5, \"pm10\": 20.3, \"machine\": { \"name\" : \"machine-one\", \"type\" : \"type-1\", \"code\" : \"mot1\" } }" \
  -d -V 5 -q 0
Enter fullscreen mode Exit fullscreen mode

The message appears in the web console (bottom of the page):

https://<AWS_DEFAULT_REGION>.console.aws.amazon.com/iot/home/#/test
Enter fullscreen mode Exit fullscreen mode

Check that the message is also stored in the DynamoDB table:

# test the data inserted by querying dynamodb
aws dynamodb query \
    --table-name $dbtable_name \
    --key-condition-expression "plant_name = :plant_name" \
    --expression-attribute-values '{":plant_name": {"S": "plant-abc"}}'
Enter fullscreen mode Exit fullscreen mode

Or navigate the web console, then select the table:

https://<AWS_DEFAULT_REGION>.console.aws.amazon.com/dynamodbv2/home/#tables
Enter fullscreen mode Exit fullscreen mode

2. Setup the gateway device and sensor

2.1 The "hard" part ;)

Start by connecting the Nova PM sensor...



...to the RaspberryPi4 gateway device via USB:



2.2 Device configuration

2.2.1 Installing configuration, user and service software module

Create a JSON configuration file (to configure the IoT client instance later on):

device_config=$(cat <<EOF
{
  "host": "$endpoint_address",
  "port":8883,
  "clientId": "$client_id",
  "thingName": "$thing_name",
  "cert": "./gateway.certificate.pem",
  "key": "./gateway.private_key.pem",
  "ca": "./AmazonRootCA1.pem",
  "topic":"$monitor_topic",
  "mockSensor":false,
  "sensorPath":"/dev/ttyUSB0",
  "pushFrequency":3000,
  "pmMultiplier": 10
}
EOF
)

# save the configuration to a file
echo $device_config > device_config.json
Enter fullscreen mode Exit fullscreen mode

Perform a cat device_config.json and cross-check that all fields have values.

From the terminal, connect to the Raspberry gateway device (assuming the gateway device already connected to your network) to copy the service and the certificate files:

cd ~/bedrock-studio-chatbot
scp -r pm-monitor-service pi@raspberrypi.local:~/
Enter fullscreen mode Exit fullscreen mode

Connect to the Raspberry:

ssh pi@raspberrypi.local
Enter fullscreen mode Exit fullscreen mode

Make the service user, config location:

# create the service user "pmmonitor" and provide a password
sudo adduser pmmonitor
Enter fullscreen mode Exit fullscreen mode

Move the files and assign the permissions:

# move the project file to the user home
sudo mv ~/pm-monitor-service /home/pmmonitor

# assign permissions
sudo chown pmmonitor /home/pmmonitor/pm-monitor-service -R

# assing permissions to read the input data from the sensor
sudo chown pmmonitor /dev/ttyUSB0 # or any USB port you have set in the device_config.json ("sensorPath":"/dev/ttyUSB0")
Enter fullscreen mode Exit fullscreen mode

2.2.2 Installing NodeJS runtime 

# install nodejs 18
curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash -
sudo apt install -y nodejs

# login as the service user
sudo su pmmonitor

cd ~/pm-monitor-service
Enter fullscreen mode Exit fullscreen mode

2.2 Run the pm-monitor-service 

# start the service
nodejs main.mjs
Enter fullscreen mode Exit fullscreen mode

Going back to web console, access the DynamoDB table "pm_values" (link), click on button "Scan" to see the data.

https://<AWS_DEFAULT_REGION>.console.aws.amazon.com/dynamodbv2/home/#table?name=pm_values
Enter fullscreen mode Exit fullscreen mode



You can stop the service at any time by:

ctrl+c
Enter fullscreen mode Exit fullscreen mode

Note: If you want the service to run automatically at boot time, you may want to create an .service file for "systemd".

Next, Coming Soon...

Configuring the Amazon Bedrock Studio Workspace.

Previous

Introduction

Image
Construtores e herança
Image
Tech Watch #1
Image
Implementing Code Coverage and Test Coverage for Quality Metrics and Software Excellence
Image
Provide context to GitHub Copilot Chat

github, githubcopilot, generativeai, ai
Image
2326. Spiral Matrix IV
Image
13 Hidden Windows Productivity Tricks You Should Know

productivity, windows, programming, devtips
Image
13 Hidden Windows Productivity Tricks You Should Know

productivity, windows, programming, devtips
Image
Top 5 Tech Trends of 2024
Image
Managing Cloud Costs and Security with Tailwarden
Image
How to Implement Singly Linked List in Python
Image
Part 9: Building RESTful APIs with Node.js and Express

node, javascript, api, express
Image
Cairo lang Hello World for JS developers
Image
ChatGPT Built Me A Linux GUI! (Tkinter)
Image
Understanding PHP Metaprogramming: Dynamic Code Manipulation
Image
A Comprehensive Guide to Developer Career Growth
Image
Comprehensive Guide to PHP Sessions: How They Work with Examples
Image
Testcontainers: Testes de Integração em Containers
Image
Amazon S3 Access Grants introduce the ListCallerAccessGrants API to control access to end users
Image
Coffee and Data: The Unbreakable Bond Between Data Analysts and Their Daily Brew
. .
Terabox Video Player