esp32, deep sleep and mqtt

I saw a lot of posts about sending mqtt messages from an esp32/esp8266 with deep sleep between the sending cycles. This works fairly well as long as you keep a some time for the message to transmit before the deep sleep cycle.
However, when you want to receive messages things get more interesting. If you just sent the callback on the esp32 to receive messages during the wake up time the messages almost never got through.
This was a bit puzzeling for me at first, as reliability was one of the main reasons why i choose to use MQTT in the first place.

My first thoughts were that the messages got lost because the esp went to sleep without proper disconnecting and the mosquitto server was trying to send with no response. So I did try sending them in QoS 1 instead (and trying to implement a more graceful shutdown before sleep on the esp side), but the effects were exactly the same.

But then I thought back about queues and topics, and that MQTT topics won’t work like queues. That was exactly the point. MQTT Topics are not retained on default. That means the mosquitto server is only sending the message to the subscribed clients when a message arrives. If the client is sleeping/unsubscribed the message will be lost.

Howerver, if you send the message in “retained” mode, like “mosquitto_pub -t esp32/output -r -m on” the message will stay on the topic in the MQTT server. So, each time a client connects it will get that retained message, for example if a relais should be turned off or on.

/* Project ESP32, MQTT and Deepsleep */
#include <WiFi.h>
#include <PubSubClient.h>

#define wifi_ssid "<ssid>"         //wifi ssid
#define wifi_password "<password>"     //wifi password

#define mqtt_server "<mqtt>"  // server name or IP
#define mqtt_user "username"      // username
#define mqtt_password "password"   // password

#define debug_topic "debug"                   //Topic for debugging

/* definitions for deepsleep */
#define uS_TO_S_FACTOR 1000000        /* Conversion factor for micro seconds to seconds */
#define TIME_TO_SLEEP 20              /* Time ESP32 will go to sleep for 15 minutes (in seconds) */
#define TIME_TO_SLEEP_ERROR 3600       /* Time to sleep in case of error (1 hour) */

bool debug = true;             //Display log message if True

WiFiClient espClient;
PubSubClient client(espClient);

void setup() {
  Serial.begin(115200);     
  setup_wifi();                           //Connect to Wifi network
   
  client.setServer(mqtt_server, 1883);    // Configure MQTT connection, change port if needed.

  if (!client.connected()) {
    reconnect();
  }
  
  client.loop();
  delay(100);
  client.loop();
  delay(100);
  client.unsubscribe("esp32/output");
  client.disconnect();
  
  esp_sleep_enable_timer_wakeup(TIME_TO_SLEEP * uS_TO_S_FACTOR); //go to sleep
  Serial.println("Setup ESP32 to sleep for every " + String(TIME_TO_SLEEP) + " Seconds");
  Serial.println("Going to sleep as normal now.");
  esp_deep_sleep_start();
}

void callback(char* topic, byte* message, unsigned int length) {
  Serial.print("Message arrived on topic: ");
  Serial.print(topic);
  Serial.print(". Message: ");
  String messageTemp;
  
  for (int i = 0; i < length; i++) {
    Serial.print((char)message[i]);
    messageTemp += (char)message[i];
  }
  Serial.println();

  // Feel free to add more if statements to control more GPIOs with MQTT

  // If a message is received on the topic esp32/output, you check if the message is either "on" or "off". 
  // Changes the output state according to the message
  if (String(topic) == "esp32/output") {
    Serial.print("Changing output to ");
    if(messageTemp == "on"){
      Serial.println("on");
    }
    else if(messageTemp == "off"){
      Serial.println("off");
    }
  }
}


//Setup connection to wifi
void setup_wifi() {
  delay(20);
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(wifi_ssid);

  WiFi.begin(wifi_ssid, wifi_password);

  int count=0;
  while (WiFi.status() != WL_CONNECTED) {
    delay(100);
    count++;
    Serial.print(".");
    /* I had some cases where the esp was stuck in connection to wifi mode */
    if (count > 100) {
      Serial.println();
      Serial.println("something bad happened, trying to reset");
      ESP.restart();

    }
  }

 Serial.println("");
 Serial.println("WiFi is OK ");
 Serial.print("=> ESP32 new IP address is: ");
 Serial.print(WiFi.localIP());
 Serial.println("");
}

//Reconnect to wifi if connection is lost
void reconnect() {

  while (!client.connected()) {
    
    Serial.print("Connecting to MQTT broker ...");
    if (client.connect("ESP32Client")) {
      Serial.println("OK");
      client.setCallback(callback);
      client.subscribe("esp32/output");
      client.loop();
      delay(500);
    } else {
      Serial.print("[Error] Not connected: ");
      Serial.print(client.state());
      Serial.println("Wait 5 seconds before retry.");
      delay(5000);
    }
  }
}

void loop() { 
}
Posted in Uncategorized | Comments Off on esp32, deep sleep and mqtt

Persisting Joda-Time Fields with openjpa

Lately i became curious about how to store joda-time objects into a database with openjpa. Well, its supprisingly easy if you only want to store/retrieve the objects:


import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.Id;

import org.apache.openjpa.persistence.Persistent;
import org.joda.time.DateTime;

@Entity
public class EntityA {
@Id
@Column(name="id_a")
private Integer id;

@Persistent
private DateTime startDate;
...
}

Thats it .. of course what happens is, that openjpa does serialize the DateTime object and stores them as a BLOB field in the database, which might not be always the preferred solution.

Another way would be to add @Externalizer and @Factory annotations to the DateTime object, and write a custom convertor that would convert DateTime to SQL Timestamp or similar.

Posted in Uncategorized | Leave a comment

Reducing Raspberri PI’s memory usage

I just got my Raspberry pi recently and found this really usefull link for tuning Raspbian to use less memory :

http://extremeshok.com/blog/debian/raspberry-pi-raspbian-tuning-optimising-optimizing-for-reduced-memory-usage/

Posted in Uncategorized | Tagged , , | Leave a comment

Git with Jenkins

Recently I did try to use jenkins together with git on windows. During configuration i stumbled along some blogpost that advised to use git.cmd instead of git.exe (which does set some more enviromnent variables, for example %HOME% to use _nettrc of curl)

Unfortunately using git.cmd leads to many very strange errors in jenkins, like missing branches, or ambiguous checkouts. Testing with some random Java project from github gave the same results, but reverting back to git.exe (and setting %HOME% for the serveruser manually) did the trick .. everything works correctly as expected 🙂

Posted in Uncategorized | Leave a comment

Hudson/Jenkins integration in Eclipse

Now, that the hudson-eclipse plugin seems dead for quite a while (not anything since 2009) i was briefly thinking about writing my own update with the xml-api by Jenkins.
But then i discovered that the mylyn project did what i wanted and even more ! With mylyn 3.5 there is an Jenkins/Hudson connector that will allow you to see the status of each job, see the test results, check the Jenkins console in eclipse, even provide a clickable changelog !

Check out the new features of http://www.eclipse.org/mylyn/new/ !

Posted in buildmanagement, java | Tagged | Leave a comment

Logback Console in Eclipse

Recently i did find there is a console plugin for logback in eclipse. Unfortunately its based on a very old version of logback (0.9.9), so i searched for the sources, bumped the logback/slf4j versions to 0.9.28 and 1.6.1 and fixed various things in the plugin. Now there’s a new version of this plugin available here, i’ll post the sources to github soon.

This version is compiled with jdk 1.6 and has eclipse 3.6 as target platform. If you’ll need something else, drop a comment ..

The documentation at http://logback.qos.ch/consolePlugin.html is still valid, just unzip the file in your plugins folder and select the logback view. After you did configure logback to use the console in logback(-test).xml things should start to show up in eclipse.
The neat thing is, that you can click on each logentry in the console to see where that specific logentry was created. Like linked stacktraces but for logfile enties.

*Update* Version 1.1.2 fixes an overlooked bug while viewing a stacktrace: logback eclipse Plugin 1.1.2

Posted in java, logging | Tagged , , , | 2 Comments

Running SoapUI Integration Tests in Hudson

Recently i’ve been wondering how to automate webservice(SOAP) integration tests. My first idea was to use soamoa, but it seems a bit buggy at the moment and the last version seems quite a while ago.

But SoapUI does offer a native maven plugin, which can run SoapUI testcases directly. So I set up a small maven project, and add a soapUI testcase like this:

<build>
 <plugins>
   <plugin>
     <groupId>eviware</groupId>
     <artifactId>maven-soapui-plugin</artifactId>
     <version>3.6.1</version>
     <executions>
       <execution>
         <id>SoapUITest1</id>
         <configuration>
           <projectFile>src/test/soapui/SoapUITest1-project.xml</projectFile>
           <outputFolder>${project.build.directory}/surefire-reports</outputFolder>
           <junitReport>true</junitReport>
           <exportAll>true</exportAll>
           <printReport>false</printReport>
         </configuration>
         <goals>
           <goal>test</goal>
         </goals>
         <phase>test</phase>
       </execution>
     </executions>
    </plugin>
  </plugins>
 </build>

Now, when i run “mvn test” from the command line, everything looks fine and the tests get correctly executed. But after I import the project in Hudson, i can’t see the test results 🙁 This seems to be a bug in Hudson, the only workaround at this point seems to create a freeform project in hudson instead of a maven2 one, add a maven2 step (mvn test), then add “**/target/surefire-reports/*.xml” under “Publish junit test results”. Now Hudson correctly does display all test results on the project page.

The only point left is to run multiple SoapUI projects from one maven project, for this one needs to add multiple <execution> tags, maybe the future versions of the soapui-maven-plugin will support easier configuration.

Posted in java, Testing | Tagged , , , | Leave a comment

Running selenium-hub inside Tomcat

The usual process of starting selenium-hub seems to be an ant goal that does run an integrated jetty server. While this is easy to run it is not so easy to run in a server enviromnent. Also, as i have an existing tomcat already running and the hub does not do that much i was wondering if i can run the server inside tomcat.

The first look was dissapointing, everything is bundeled in jar files, no war file in sight. Fortunately selenium-hub is just a bunch of servlets with some logic behind, so it should be easy to add some maven scripts to create a war file.

Selenium-hub also depends on selenium-core, which is also included in the selenium-hub distribution. So i created a fork of selenium-grid on github and add some maven scripts. You can see the results here: https://github.com/mglauche/selenium-grid . Right now only the core and the hub part are mavenized.

A complete war-file can be found here: selenium-core-hub-1.0.8-SNAPSHOT

Posted in Testing | Tagged , , | Leave a comment

Introduction to Scala – Part 1

What is scala ?

Scala is a relativ new compiled programming language that takes a lot of ideas from python, ruby and others. From the scala homepage:

Scala is a general purpose programming language designed to express common programming patterns in a concise, elegant, and type-safe way. It smoothly integrates features of object-oriented and functional languages, enabling Java and other programmers to be more productive. Code sizes are typically reduced by a factor of two to three when compared to an equivalent Java application.

Why do we need another programming language ?

I mean, there are a wealth of different languages that created quite a buzz recently like ruby, python, c#. Why do you need scala ? For one, scala is a compliled language unlike ruby and python. The scala compliler generates java-bytecode that is compatible with the java jre.
This looks like a small feature but it is really one of the most interesting features of scala:

Seamles Integration in java

You can directly use java objects in scala and vice versa. Scala directly compiles to java .class files. You can deploy scala applications to your J2EE container. Which means if you are a java shop you can use your existing tools and servers to deploy scala applications. No need for a new application server, JBoss, websphere and others will work fine.

This is a very important point for larger companies. Changes in these companies can only happen graduately, so you can keep your existing knowledge and workers. The server administrators still only need to maintain the same application server they’ve been knowing for years. You even can continue to use your own libraries you’ve developed in java over the years. These points are very important if you want to make a successfull migration towards scala.

Why use Scala over Java ?

If scala is fully compatible with java, why use it ? Why learn a new language when the old one seems good enough ?

  • Scala corrects many of javas mistakes
    • Everything is an Object in scala
    • Type inference -> only need to declare the type on the right side of the assignment (i.e. var dummy= Map[String,Int]() )
    • True Mixins
    • Scala is both, a object oriented and a functional programming language
  • Fresh start
    • Clean Language, nothing feels like an addon (like generics for example)
    • much more compact, like no need for getters and setters
    • No semicolons

Hello World in scala

Enough of the theory. How does a typical scala program look like ?

object HelloWorld {
  def main(args: Array[String]) {
    println("Hello, world!")
  }
}
Posted in scala | Tagged , , | Leave a comment

zlib.dll not found when installing the current ruby-win32

Missing libraries

zlib.dll not found

After downloading the latest win32-releases of ruby i was greeted with two error messages, SSLEAY32.dll not found and zlib.dll not found. The SSL library is kinda optional (gem, etc basicly work without it), but zlib not.

For this, the zlib library is needed, luckily there’s a native win32 port of it. So, download the latest win32 dll from zlib.net,
extract the zlib1.dll, rename it to zlib.dll and place it into the ruby bin directory.

SSLEAY32.dll not found

For SSLEAY32.dll the safest way i’ve found seems to use the version from the win32 postgresql client, which can be found here.

Just copy SSLEAY32.dll and libeay32.dll from the postgresql/bin directory into the ruby bin (or actually any other directory which is in your path)

readline.dll not found

Now you think everything is working, and start irb for the first time. Dang. readline.dll is missing. Readline win32 binaries can be found at http://gnuwin32.sourceforge.net/packages/readline.htm. Download the latest win32 build and copy the readline5.dll from the bin directory to your ruby bin directory again. Don’t forget to rename it to readline.dll.
Now everything should be working smoothly.

Posted in ruby on rails | Tagged , , | Leave a comment