SAWA – Real-Time Sensor Platform

 

Experience: http://linux-utils.blogspot.com/2011/01/arduino-remote-control-outlet.html

Platform features, summary

Web application

- create networks
- create sensors
- create actuators
- add members to a network
- call functions or commands in actuators
- define policies
- define rules
- define notifications
- browse networks and sensors in Google Maps
- Visualize data charts, in average or raw mode
- delete and unsubscibe XMPP nodes
- check event log
- export sensor data in multiple formats
- publish sensor data via HTTP POST
- see users in system (admin users)
- edit system configuration online (admin users)
- check services status (admin users)
- add images to network and sensor galleries
- add properties to sensors
- select chart type

sox_tools

sox_tools is a utility command line XMPP client built for interacting with an XMPP server. Most of its functions is related to PubSub protocol (XEP 0060). The tool help explain each option as shown.

Available options:

	--user			Select user from config

	listen			Listens for pubsub events
	commands         	Listens for commands and call apropriate action in actuators
	publish			Publish to a pubsub node
	publish_cmd		Publish a command to a pubsub node
	last_item	    	Get last item from a pubsub node

	create			Creates a pubsub node
	subscribe	    	Subscribe to a pubsub node
	unsubscribe		Unsubscribe from a pubsub node
	add_member		Add a member to a pubsub node
 	subscriptions		List pubsub subscriptions
	affiliations		Lists pubsub affiliations
	delete			Deletes a pubsub node

	create_user		Creates a XMPP user
	delete_user		Deletes a XMPP user

Screenshot of database

Wine monitoring

“To gain scientific understanding into the vinification process of Madeira wine, we intend to use sensors to study various parameters of the process (e.g. Ethanol, pH, total acid, volatile acid, density, fixed acidity, ash,), as well as actuators to control the environment (e.g. temperature, humidity, etc.)”

Search for sensors

One practical use case was wine monitoring. Due to our limited knowledge in electronics, we looked towards sensors with easy interfacing, to be able to connect in Arduino. Here is a list of  measurable parameters that can be interesting for wine analyses, however, many of these parameters are manually analysed in the lab, using heavy equipment.

After some search, we found some experiences using gas sensors and other sensor, which helped us to make a decision to order or not.

The main criteria for selecting sensor for this experience was :

- easy interfacing
- no maintenance
- affordable

Measure in liquid	Order	Use
pH	Yes	(+) Seems reasonable
Temperature	Yes	(+) Reasonable
Color	No	(-) Expensive(-) Specific HW and SW
Dissolved Oxygen	No	(-) Expensive(-) Specific HW and SW(-) Maintenance
Volatile acidity	No	(-) No sensor
Total acidity	No	(-) No sensor
Oxidation	No	(-) No sensor
Ethanol	No	(-) Expensive(-) Specific HW and SW(-) Maintenance
SO2	No	(-) No sensor

Measure in gas	Order	Use
CO	No	(+) Reasonable(-) Less important in wine
CO2	Yes	(+) Reasonable
Alcohol	Yes	(+) Reasonable
Relative Humidity	Yes	(+) Reasonable

pH – pH probe from Vernier

Temperature – Temperature sensor with steel head, from Seeed Depot

CO2 – Parallax CO2 module with mg811 c02 gas sensor, from Parallax

Alcohol – MQ3 gas sensor, from SparkFun

RH – Relative Humidity sensor, from SparkFun

Testing sensors

After the arrive of ordered sensors, we started to test them.

Building probe

The probe construction was the most difficult part to do, because of excess of wiring and lack of space inside the probe. Three attempts were made to build a stable probe (float with equilibrium in the wine), but it was very difficulty to wire and use all sensors along with pH probe in the same module. And in the end the probe was not floating :’(.

Building probe with success

After the previous experience we came with a more simple solution, use 2 modules, one for pH and temperature, and the other for gas sensors, the last attached to the top of the wine container.  The upper module has 2 separated parts, the top one contains the micro-controller (Arduino) , DC jack and RF transmitter an receiver. The bottom part contains the c02, alcohol and relative humidity sensors.

Installation

We have the probe uploading sensor data every minute, (RF 433MHz module). The sensor gateway (Arduino +  ethernet shield + RF receiver module), receives this information and sends via HTTP POST to server, which then is published using XMPP protocol.

EOF =)

Last update to system

New updates to system, and to sawa wiki

WEB
removed flexigrid
added better notifications
added raw chart
changed all files to the right names (networks, sensors, etc)
better way to upload sensor data via HTTP
added sensor gallery
options to select data parameters on exporting data
edit configuration
services status
user list

DATABASE
added support for messages, events, sensor properties, custom rules and policy groups

SERVICES
added support for date rules and database history rules
added events logging support

Storage statistics, update rates

We have 327.735 records in database, with total size of 27.1 Megabits, which gives an average of 10.84 bytes per record in database.

3552051.2 (bytes)/327735 (records)  = 10.84bytes per record

So when selecting the upload rate for sensors, we should keep in mind this values.

Time to fit one GigaByte
Interval	Records	days		years
Second	1	1146.65		3.14
	10	114.66		0.31
	100	11.47		0.03
	1000	1.15		0
Minute	1	68798.82		188.49
	10	6879.88		18.85
	100	687.99		1.88
	1000	68.8		0.19
Hour	1	4127929.15		11309.39
	10	412792.92		1130.94
	100	41279.29		113.09
	1000	4127.93		11.31

If you have a sensor with the upload rate at 1 seconds, in 3.14 years the sensor data will take 1GB of space

Reliability/Performance tests

in the past week i have been testing sensors for the wine monitoring use case, since our pH probe broke, i am waiting for a new one to arrive and a humidity sensor. This probe will have: c02, alcohol, RH (relative humidity), temperature and pH sensors. Maybe in next week i will create a post about the probe. Until then, i have returned to my report, starting with tests. For now i completed Reliability and Performance tests for publishing data and notifications.

Results:

Since the fail rate was to high in POST, i found the problem was related to C command line tool “sawa_publish”, which publishes the POST data to XMPP.

pthread_mutex_lock.c:87: __pthread_mutex_lock: Assertion `mutex->__data.__owner == 0′ failed.

After looking at google, i did not find any usefull/clear solution, so i decided to simplify the publish function and repeat the same process when fails.

- added a new publish function

—- removed variable and data arrays

—- added repeat process when fails, up to 3 times

	function publish2()
	{

		$this->form_validation->set_rules('USERNAME', '', 'trim|required');
		$this->form_validation->set_rules('USERPASSWORD', '', 'trim|required');
		$this->form_validation->set_rules('NODE', '', 'trim|required');
		$this->form_validation->set_rules('VARIABLEID', '', 'trim|required|numeric');
		$this->form_validation->set_rules('DATA', '', 'trim|required');

		if($this->form_validation->run()){

			$res=0;
			$count=0;

			while(!$res && $count <=2){
				$res=$this->sox_wrapper->publish(
								$this->input->post('USERNAME'),
								$this->input->post('USERPASSWORD'),
								$this->input->post('NODE'),
								-1,
								-1,
								$this->input->post('VARIABLEID'),
								$this->input->post('DATA')
				);
				$count++;
			}

			if($res)
				echo "OK\n";
			else
				echo "FAIL\n";

		}else{
			echo "FAIL\n";
		}

	}

Bash script example to upload data via HTTP POST (useful for resource limited devices)

#!/bin/bash

# settings
USERNAME=user123@hci.uma.pt/scripts
USERPASSWORD=pass123
NODE=testnode
VARIABLEID=23

URL="http://dev.hci.uma.pt/sawa/xmpp/publish2"
DATA=1

for ((  i = 1 ;  i <= 10;  i++  ))
do
  DATA=$i;
  curl -X POST --silent --data-binary "USERNAME=$USERNAME&USERPASSWORD=$USERPASSWORD&VARIABLEID=$VARIABLEID&NODE=$NODE&DATA=$DATA" $URL
done

Updated table with new code changes

Intelligent Mail Box

What about you being  notified when you have mail ? by email or sms, or other type…

I set up this intelligent mail box with 2 sensors and one transmitter.

Magnetic door switch $2.00

Mail sensor ( i invented it , works 1000 times better than distance sensor, or other you may use).

RF transmitter Kit for $4.90, you only need the transmitter for this.

Arduino/Compatible from $23

Comparing the size of RF transmitter module with an old Nokia

To connect any switch,  im using these “2pin plugable terminal module” sold by Seeed Depot, which is very useful.

Here is the mail sensor, basically its a simple switch, with two separated metal parts, they are very flexible in order to make contact when a new mail touches.

Mail box, red would be better

I pasted one part of magnetic door sensor to the box, and the magnet part on lock, so when we turn the key to open door, this magnet will go up.

Adding the box with arduino, and connected sensors.

Here we can see mail sensor attached to mail box and a external power supply.

Programming arduino..

Creating a new policy one server, this policy has only one filter, when new_mail value equals 1 then the action is triggered, which in this case was a SMS notification.

New mail arrived!, check out the video

Video
Video

Testing Sensors 2

Testing sensors 1

System use cases and refrigerator experience

System overview

SAWA can be considered as a platform, with a set of services and web application. It uses XMPP Protocol to transport all sensor data in real-time, and allowing secure connections. Using XEP-060, Publisher-Subscribers paradigm it is possible to share data with multiple users with flexibility and scalability.This project uses soxlib with modifications from Sensor Andrew project.

Project blog, Project source

Web application

In web application is possible to manage networks and sensors, manage XMPP nodes, define policies, and share data.

-Technologies

Google Maps, Flexigrid, Open Flash Charts 2, jQuery, Ajax, JavaScript, Openfire Css, CodeIgniter, PHP, C, MySQL

Services

-Datarecorder

This service is responsible to store sensor data in database, when published in nodes

-Actionchecker

This service is responsible to match all sensor data with existing policies, and launch these actions when policy matches.

Each notification type is included as plugin, making this service more easy to extend.

-Scheduller

This service is responsible to order the previous services to subscribe or unsubscribe from PubSub nodes.

-Technologies

glib-2, gmodule, libloudmouth, soxlib, mysql, C, XML, XMPP

Servers

Openfire XMPP Server

MySQL database server

Apache HTTP Server

Postfix email server

Tools

install soxlib and sox_tools

soon

sox_tools

-sox_tools config

create .sox_tools.cfg in your home

[XMPP]
PORT=10223
HOST=hci.uma.pt
PUBSUB_SERVER_NAME=pubsub.hci.uma.pt
TIMEOUT=10

[DEFAULT_USER]
JID=user123@hci.uma.pt/resource_xpto
PASSWORD=user123

- in this example we are publishing the value 25 with unit “C” from variable 14 in user123_network network

sox_tools publish user123_network 14 25 C

Bash script (HTTP method)

#!/bin/bash

# user credentials
USERNAME=user123@hci.uma.pt
USERPASSWORD=user123
NODE=user123_network
VARIABLEID=55

# publish Url
URL=http://dev.hci.uma.pt/sawa/xmpp/publish

# data value
DATA=99

curl -d "USERNAME=$USERNAME&USERPASSWORD=$USERPASSWORD&VARIABLEID[]=$VARIABLEID&NODE=$NODE&DATA[]=$DATA" $URL

How to use this system?

Create account

After creating an account, login with your credentials.

Create Network

When creating a new network it is possible to specify the access level, and network location.

Create Sensor

To create a sensor, we have to select the network which will belong to, the type (sensor or actuator), location and the variables of the sensor (this variables can be for example door status, open time and closed time count from a door switch sensor). After this steps, we are now able to upload data and see them on charts.

Publish data

There is two methods of sending data, one is HTTP POST, useful for simples devices, without capabilities of parsing XML and establishing SSL connections. The other method is using sox_tools, a specific XMPP client for this system. Last one has the ability to publish, listen events and manage pubsub nodes.

using XMPP Client ( sox_tools recommended )

sox_tools publish myHome 11 14698

using HTTP POST

curl -d -m 10 “USERNAME=user123@hci.uma.pt&USERPASSWORD=user123&VARIABLEID[]=11&NODE=myHome&DATA=14698″ $URL

View data charts

All sensor data is set to be recorded by default, in “Data” menu, select the chart type and sensor variables to be represented, and to finish click on interval to generate graph.

Create policies

In policies we have, four types of notifications, email, sms, url and publish events to actuators, in this example I created a policy to alert me when thee door is more than 300 seconds open. I repeated the same test with all notifications types, the results are bellow.

Email notification

Email notification working

URL notification

Datarecorder service output, policy Url being called

SMS notification

SMS notification arrived

Node event

Event published on node

Practical use cases

Monitoring a refrigerator

I used this experience to test the system, so the objective was to record sensor data.
The variables I selected to record was: door status(open or closed), time open (number of seconds the door was open), and temperature.
Since arduino is very resource limited, i used HTTP POST method to publish data.

Sensors

• sensors
  • 1 temperature sensor
  • 1 photo-resistor

• devices
  • 2 arduinos
  • 1 ethernet shield
  • 1 RF kit

Network diagram

Sensor Network diagram

- The arduino on refrigerator sends messages to gateway (arduino in router) with format msg_id:sensor_variable:value (example: 455:door:1:temperature:10)
- Then gateway sends via HTTP POST the sensor value with user credentials and sensor identification
- Then on server, this data is published to XMPP node

Pictures

Refrigerator with Arduino

Arduino with RF transmitter module (with yellow wire as antenna)
Blue wire is energy source, red, black and white are connected to sensors

Instead of having a door switch, I used a photo-resistor as door switch, which measures the amount of light. So when we open the door, the light from refrigerator will go up and go down when door is closed.

Here is two basic sensors the “door switch” and temperature sensor

My sensor gateway, responsible for receiving data through RF and send to server

Arduino+Ethernet Shield+RF Reciver

Data charts

Possible experiences

Inteligent Mail box

adding soon

Wine monitoring

adding soon

Energy savings

turn power off from all standby devices, if inactivity in a room is greater than a number of seconds

arduino or pc+PIR+Relay

GPS

create policy with geographical limitations (square), when device enters this area an action will be triggered