rewrite backports

Introduction

From the last weekend of july i've started to backport features from the new rewrite code to development. For several reasons.

1. I have been running the rewrite code for some time now, but haven't finished it enough to replace the current development code. So i consider 99% of the code to be stable.
2. I still want to test this new code in public to test if my assumptions are right.
3. I don't want to keep these new cool features away from the general user any longer.

This blog is meant to inform you all about the changes that are slowly being backported while development on the rewrite code is still going on.

New core changes already backported

All the changes below are already being backported from rewrite. This however means that the new code has been implemented besides the old code, until all the old code has been removed.

Internal event driven actions

All pilight code was intertwited in one big knot. Much like an old telephone central. When new module was written, a cable had to be plugged in somewhere in another older piece of code.

The new event driven approach overcomes this by implementing a producer/consumer pattern. So the config update functionality now triggers the REASON_CONFIG_UPDATE event. On the other side various parts of pilight react on this event. So, when new modules are released, they only have to listen to certain events. The code that triggers the events doesn't have to know that someone is or is not listening.

Threadpool

Since the beginning of pilight, all parallel code was run in a separate thread. This could lead to over 20 threads in complex setups. The new pilight code uses a threadpool. Functions are added to the threadpool awaiting execution and a fixed number of thread (workers) will execute them in parallel. At the moment pilight uses a fixed number of 4 threads to handle all the work.

Centralized socket handling

Since the beginning of pilight, all sockets communication was implemented on its own. Because, pilight uses non-blocking sockets, each implementation also used its own code to wait for events. The new approach uses a centralized socket handling library that processes all sockets at once. So all seperate threads were reduced to just one, which grealy reduces resources.

Custom webserver implementation

The latest webserver implementation used by pilight (Mongoose) had some big limitations:
- SSL could not be easily ran on mbedtls but only on OpenSSL.
- Adapting Mongoose to use a centralized socket pool was quite impossible.
- Adapting Mongoose to use the threadpool was quite impossible.
- Mongoose had a big overhead in functionality already present in pilight.

I thereby decided to drop Mongoose and created my own webserver (by reusing some of Mongoose and others code). This new webserver perfectly fits the new pilight codebase and also implements SSL by default.

All backported code has already been actively unit tested

All new code written is being tested constantly to be sure no new bugs are introduced when writing new code. The status of this check can be seen on the rewrite code page:

https://github.com/pilight/pilight/tree/rewrite

What's changed

28-07-2017

At this moment the https, mail, and webserver module and the full pilight-sha256 program has been backported from rewrite. The asynchronous I/O library libuv has been added as well as the new SSL and eventpool module. The openweathermap and weather underground protocols have been adapted to use this new code as well as the pushbullet and pushover event actions.

• pilight now supports a HTTPS webserver which can be configured in the settings:

  'webserver-https-port': 5002

• pilight does not try to discover the GPIO platform it is being ran on (like Raspberry Pi, Hummingboard, Odroid) anymore, but wants you to explicitly tell this to pilight. This can be done using the following setting:

  'gpio-platform': 'raspberrypi2'

  The platforms supported can be retrieved through:

  pilight-daemon -H

• wiringX and mbedtls are not staticly compiled anymore and therefor require seperate installtion. So make sure you have the stable repository enabled (besides the nightly).

  deb https://apt.pilight.org/ stable main
  deb https://apt.pilight.org/ nightly main

• pilight also stopped detecting if the mailserver you have configures requires an SSL connection. To tell pilight about the SSL requirement of a mail server a new setting has been added. Servers that switch from a plain connection to SSL require a 0 value here.

  'smtp-ssl': 1

• The ram resource usage has been removed, because this wasn't reliable.

12-08-2017

This iteration includes the backport of the eventing code. This involves the following changes:

• Rule can be written loose-typed:
  
  Old behavior were we needed the IS operator because the comparison was non-nummeric

  IF switch.state IS on THEN ...

  New behavior is that we don't have to worry about this anymore and can just use the == operator
  

  IF switch.state == on THEN ...

• Removal of the IS operator
  Due to the above reason the IS operator has been deprecated.
   
• Labels can be used inside rules
  Also due to the above reason labels can be used inside rules. Previously, this was impossible because labels were already loosely typed, while the eventing library wasn't.
   
• Better quote parsing
  This change makes it easier to write rules and not having to worry about quoting.
  
  Old behavior
  

  DATE_FORMAT(DATE_ADD(date, +1 HOUR), \\\\\\\\"%Y-%m-%d %H:%M:%S\\\\\\\\", %H.%M)

  New behavior
  

  DATE_FORMAT(DATE_ADD(date, +1 HOUR), %Y-%m-%d %H:%M:%S, %H.%M)

03-09-2017

Various API protocols have been backported in this iteration and the new functionality which allows event operators to be written in Lua.

• 433Mhz UUID
  It should now be possible to use the UUID value for 433.92Mhz devices to tell which node should or should not broadcast.
   
• Lua operators
  Event operators are now written in Lua and therefor can be extended much easier. You can find them at /usr/local/pilight/operators. This adds a dependency on the liblua5.2-0 library.
   
• New webserver location
  The webserver is now installed in /usr/local/pilight/webgui. If your config points to the old location either remove that setting or update it.