Since I first learned of rsync, copying files have not been the same. If you do not know rsync, you should read up on it now.

Though I like rsync I have always been a little bothered that there was seemingly no way of forcing rsync to count the total number of files before transferring them, thus making the --progress option partially useless. This time I did some research, and found that if your rsync is newer than version 3.1.0 there is an --info option.

From the rsync manual page:

This option lets you have fine-grained control over the information output you want to see. An individual flag name may be followed by a level number, with 0 meaning to silence that output, 1 being the default output level, and higher numbers increasing the output of that flag (for those that support higher levels). Use --info=help to see all the available flag names, what they output, and what flag names are added for each increase in the verbose level. Some examples:

    rsync -a --info=progress2 src/ dest/
    rsync -avv --info=stats2,misc1,flist0 src/ dest/

Note that --info=name's output is affected by the --out-format and --itemize-changes (-i) options. See those options for more information on what is output and when.

This option was added to 3.1.0, so an older rsync on the server side might reject your attempts at fine-grained control (if one or more flags needed to be send to the server and the server was too old to understand them). See also the "max verbosity" caveat above when dealing with a daemon.

With the --info=progress2 i get the following kind of output:

receiving incremental file list
            0   0%    0.00kB/s    0:00:00 (xfr#0, ir-chk=1001/71529)
            0   0%    0.00kB/s    0:00:00 (xfr#0, ir-chk=1000/71612)
            0   0%    0.00kB/s    0:00:00 (xfr#0, ir-chk=1000/73443)
Documents/ebooks/Humble bundle/
            0   0%    0.00kB/s    0:00:00 (xfr#0, ir-chk=1014/73485)

Which seem what I have wished for.

Lately I have had the need for a device to do some simple logging. After some brianstorming I made this list of features:

  • Battery operated.
  • Self contained.
  • Log data readable wirelesly.

I came to the conclusion that the events I wanted to log, were on/off type events, like logging whenever my doorbell is rung. Using an ESP8266 module (ESP12), to log the events to the flash memory on the module, seemed feasible even using battery power, since the module only needed power when an event triggered it. To keep track of the time, when the ESP8266 was off, a real-time clock was needed.

  • Use the ESP8266 as microcontroller, log storage and wireless interface.
  • Use a DS3231 RTC module to keep the time.
  • Leave the RTC running at all times.
  • Power the ESP8266 only when an event needs to be logged.

The ESP8266 firmware performs the following tasks. GitHub

  • Write a log entry whenever power is applied. The log entry is written to a ring buffer, ensuring that the data will not overflow. If the flash memory is full, the firmware starts overwriting the oldest entries.
  • Try connecting to a predefined access point. When logging at some remote connection, simply create the access point using the phones WIFI hotspot feature.
    • Update the ESP8266 clock and the RTC from an NTP server.
    • Create a WebSocket server. This serves the log to the phone, and enabled configuration of the predefined access point.
    • Delete the logs from flash memory.

The client is Android specific, I have no I-devices (except an Ipod), and Microsoft, well ya'know... GitHub

  • Download the log from the WakeLog device.
  • Export the log (Google spreadsheet?).
  • Configure the access point used by the WakeLog hardware.

Sometimes you end up with a system having files modified in the future. At one time this happened to my Raspberry Pi, who was set up to sent me an email when something was wrong. After a friendly reminder every hour, I set out to find the files, and change their date to something sane.

Using find, this give a list of the files:

find . -newermt "5 days" -ls

Combine it with touch , to change the date to the current.

find . -newermt "5 days" -ls -exec touch -c {} \;

This setup is used on a ECM3610 Debian machine serving as a firewall, DHCP server, and DNS server. The following assumes that you have a working installation something like this:

  • eth0 is connected to the local ( network.
  • eth1 is connected to the internet.
  • Shorewall is installed and configured for regular firewall duty.
  • dnsmasq is installed and configured to act as DHCP and DNS server.
  • lighttpd is installed and only exposed on the local network.

Functional description.

This setup allows requests to a list of domains to be redirected to an internal web server. Use this setup to block for instance known tracking servers on the internet.

Somewhere along investigating how to do this I read, in the ArchWiki privoxy page that blocking trackers creates a unique browser signature. I have absolutely no reason to doubt this, but there were some servers that I simply did not want to talk to.

Using dnsmasq to redirect domains.


If your dnsmasq is configured to listen on an interface, reconfigure it to use addresses to make stuff prettier when assigning another IP address to the interface in a little while.


Use files in /etc/dnsmasq.d/ as configuration files. Uncomment or add the following:


Get a list of servers to block, spits out a list formatted for dnsmasq, if you ask it to. The list uses as target, replace it with the address of a local web server if you want to inform the user that something was blocked, in this case.

curl -s '' | sed -r 's/' > /etc/dnsmasq.d/block2local

This could be made into a cron job to update the list from time to time.

Setting up a web server to show an information page.

I have lighttpd serving some pages to the local network, so I add another IP address to eth0, and configure lighttpd to serve this address from another directory.


Add this:

iface eth0 inet static


Add this:

$SERVER["socket"] == "" {
   server.document-root = "/var/www/blocked"

Add something that serves your purpose to the server.document-root directory. Simplest is something like:


<title>Server blocked.</title>
<p>This server is blocked on this network.</p>

Since I began programming the ESP8266, I have been struggling to get the TCP network API of the SDK working consistently without crashes. I think that I have cracked it at last. Here are some notes, so that I may never forget, and maybe help someone else.

ESP8266 TCP server.

This post is about using the ESP8266 as TCP server, but the information is useful if you want to know how to tackle ESP8266 networking in general. First lets set up a server like the SDK tells us to do it:

  • Initialise struct espconn parameters according the protocol type you want (ESPCONN_TCP).
  • Register connect callback and reconnect callback functions.
  • Call espconn_accept to start listening for packets.
  • When a connection has been made on the listening connection, the registered connect callback is called.

Some code to do this:

bool ICACHE_FLASH_ATTR tcp_listen(unsigned int port)
    int ret;
    struct espconn *conn;

    //Get some memory for the connection information.
    conn = os_zalloc(sizeof(struct espconn));
    if (conn)
        //Set up TCP connection configuration.
        //TCP connection.
        conn->type = ESPCONN_TCP;
        //No state.
        conn->state = ESPCONN_NONE;
        //Set port.
        conn->proto.tcp->local_port = port;

        //Register SDK callbacks.
        espconn_regist_connectcb(conn, tcp_connect_cb);
        espconn_regist_reconcb(conn, tcp_reconnect_cb);
        espconn_regist_disconcb(conn, tcp_disconnect_cb);

        ret = espconn_accept(conn);
        if (ret != ESPCONN_OK)
            os_printf("Error when starting the listener: %d.\n", ret);
    os_printf("Could not allocate memory for the listening connection.\n");


The SDK network API is event based. The way this works is that you register some functions for the API to call when certain events happen (connected, disconnected, data received, and so on). Your callback function receives a parameter called arg, from the SDK, which is a pointer to an espconn struct (defined in espconn.h in the SDK). This struct contains the connection information. You will probably need a way to identify a connection as it triggers the callbacks. I have additional data from my own application coupled to the connection, that I need to handle when an event happens to the connection. In fact I have a list of all open connections using my own data structures. When I first started using the ESP8266 I ignored the part of the Programming Guide, that told me to not use the arg pointer from the callbacks, to distinguish one connection from another. The following is an example of a struct I was using at the time to keep track of connections.

struct tcp_connection
    //Pointer to the struct espconn structure associated with the
    struct espconn *conn;
    //Pointer to the data meant for the current callback.
    struct tcp_callback_data callback_data;
    //Is the connection sending data.
    bool sending;
    //Are the connection closing.
    bool closing;
    //A pointer for the user, never touched.
    void *user;
    //Pointers for the prev and next entry in the list.
    struct tcp_connection *prev;
    struct tcp_connection *next;

When ever the program entered a callback, the code would compare the arg pointer to the one stored in the conn member of tcp_connection, and couple my private data to the connection that way. This works most of the time, but fails in couple if cases.

  1. The SDK tells us not to do this, which in my book tells me, that sometime later the SDK may change internally enough, that this could stop working.
  2. The reconnect and disconnect callbacks gets an arg parameter passed, that point to the listening connection, not the one that the event happened too.

There are ways around the second problem that works quite well, but why bother when the whole thing may break with some later version of the SDK, and there is a nicer option outlined in the Programming Guide?

The solution from the Programming Guide is to use the remote address and port to identify each connection. This took a little more code, but when done, revealed a few things about the flow of the connections in the API.

Connection flow in the ESP8266 API.

Let us look at some excerpts of debug log files from my HTTP server:

Listening connection.

Just after the espconn_accept call the situation looks like this:

Connection 0x3fff3158 (0x3fff3480) state "ESPCONN_LISTEN".
 Remote address
 SDK remote address
1 connection.

This is the listening connection. This connection is the one that the remote computer connects to, and has to remain open as long as the server is expected to serve connections. 0x3fff3480 is a pointer to the struct espconn data for the connection. 0x3fff3158 is the connection data, that I am keeping. The struct looks somewhat like this:

struct tcp_connection
    //SDK connection data.
    struct espconn *conn;
    //Remote IP.
    uint8 remote_ip[4];
    //Remote port.
    unsigned int remote_port;
    //Local IP.
    uint8 local_ip[4];
    //Remote port.
    unsigned int local_port;
    //Pointer to the data meant for the current callback.
    struct tcp_callback_data callback_data;
    //Pointers to callback functions for handling connection states.
    struct tcp_callback_funcs *callbacks;
    //Is the connection sending data.
    bool sending;
    //Is the connection closing.
    bool closing;
    //Pointers for the prev and next entry in the list.
    struct tcp_connection *prev;
    struct tcp_connection *next;

As will later become clear, espconn may be deallocated by the SDK. Any data that you may need to preserve to keep track of the connection should be copied somewhere else, hence the IP addresses and ports in the above struct.

New connection.

Next up someone is connecting to the server:

TCP connected (0x3fff3cb8).
Connection 0x3fff3158 (0x3fff3480) state "ESPCONN_CONNECT".
 Remote address
 SDK remote address
1 connection.

0x3fff3cb8 is the address of a new espconn struct, that the SDK has created for the incoming connection. This is where I create a new tcp_connection struct, and add it to the list of connections. The Remote address is the one saved when the connection was created, and the SDK remote address is the one currently in the espconn struct.

Data received.

A request comes in:

TCP received (0x3fff3cb8).
Connection 0x3fff3158 (0x3fff3480) state "ESPCONN_CONNECT".
 Remote address
 SDK remote address
Connection 0x3fff3d18 (0x3fff3cb8) state "ESPCONN_READ".
 Remote address
 SDK remote address
2 connections.

The callback gets a pointer to the newly created espconn struct, but still according to the docs we can not expect this behaviour.

Sending the answer.

To answer the request, find the connection that has a remote address of and use the espconn pointer (0x3fff3cb8) when calling espconn_send. After calling espconn_send you have to wait for the sent callback before sending any more data. I have not seen any mention of the data size limit of espconn_send in the official docs, but I have seen 1440 bytes mentioned elsewhere. You may have to split your data and do more than one espconn_send, wait, sent callback cycle.

This is what the situation looks like when entering the sent callback:

TCP sent (0x3fff3cb8).
Connection 0x3fff3158 (0x3fff3480) state "ESPCONN_CONNECT".
 Remote address
 SDK remote address
Connection 0x3fff3d18 (0x3fff3cb8) state "ESPCONN_CONNECT".
 Remote address
 SDK remote address
2 connections.

This is where the penny finally dropped for me. Notice the SDK remote addresses? It may be obvious, but the listening connection tracks the state of the currently sending connection. It looks like listening connection does the actual sending.

Closing the connection.

What happens when the connection is closing, or an error (reconnect) occurs, makes sense after seeing the above, but eluded me for a long time.

TCP disconnected (0x3fff3480).
Connection 0x3fff3158 (0x3fff3480) state "ESPCONN_CLOSE".
 Remote address
Connection 0x3fff3d18 (0x3fff3cb8) state "ESPCONN_CLOSE".
 Remote address
2 connections.

The SDK is a harsh mistress, and deallocates the espconn connection data before calling the disconnect/reconnect callback. That is why there is no SDK remote addresses and also why I saved the remote address in my own connection data earlier. The espconn pointer returned in the arg parameter of the callback still contain the remote address of the closing connection (and points to the listening connection). The code uses this to look up the connection and deallocate my stuff.

Calling SDK disconnect.

According to the SDK docs, rather than calling the disconnect function from a network callback function, it should be invoked through a task. My code has mostly worked without doing this, but it will give the SDK time to do some processing in between the callback and the disconnect, and I am going to try this in time.

The first thing to do if you have manifest errors, is to find out why! Do not blindly recreate your manifests.

After several syncs emerge kept complaining about missing digests for a huge amount of ebuilds on the system. I guess this shuold solve itself at some future sync, but in the meantime I couldn't merge anything because of the mising digest, so I digged around and crafted this command to rebuild all digests/manifests in the portage tree.

find /usr/portage -type d -exec sh -c 'find "{}" -maxdepth 1 -type f -name '*.ebuild'| sort | head -n 1| xargs -r -I --  sudo ebuild -- manifest' ";"

The command finds the first .ebuild file in every subdirectory of /usr/portage and runs sudo ebuld minifest on it.

...And why you shouldn't.

manifest files contain digest and file size of every source tarball that Gentoo uses to build your system. If every manifest is regenerated locally, maliciously changed source tarballs (and we just don not know for sure if there are any, do we) on you computer will get a valid manifest file, making it possible to install it.

Following up on the general notes, here I some notes on programming for the ESP8266 in C. I use Linux for development, but most of this is OS independent.


  • Prefixing a function with "ICACHE_FLASH_ATTR" places it in flash, not doing so places the function in RAM.
  • The IoT SDK is single tasked and event driven. I've always found event driven programming a pain, but this implementation stinks. There is no way to return to the system, in the middle of processing, to not stall the system tasks. The docs, says that you not may stay in a function for more than 10 ms, without the system code getting behind. If I were to start over I would try the FreeRTOS SDK, unfortunately it seems that Espressif is giving it less love.


  • Some basic code for connecting to a WIFI. GitHub
  • The connect callback returns a pointer to the struct espconn in the arg parameter. It is not the same as the one used by espconn_accept.
  • The disconnect_callbackreceives a pointer to the listening connection, not the active connected one, at least when in TCP server mode.
  • In struct espconn, member reverse, seems to be free to use., except in the disconnect callback, where it suddenly has a new value.


  • It seems the file system of choice is best placed after the code in flash.
  • On suggestion from Dave Hylands, I use a ZIP file as file system image. The file is uncompressed and used as a simple container.

Generated on 2018-05-03 01:14:21.831007