IoT status, as I see it
For the last 5 years, I have worked in a small company that was creating a software to manage data coming from various sources, and that includes IoT devices (but not only).
This is a weird ecosystem.
One would think that this was a booming one, since it was all around the IT news in the past 10 years (a bit less now), but it was mostly hype.
So what is it about, from my little corner of this ecosystem?
This post is certainly not going to provide an in depth analysis of the ecosystem, it's just a feedback of 5 years to tangent this area...
IoT, as I see it used
It was mostly about installing energy sensor (current sensors, temperature sensors, etc). Mostly for the industries, because they were under heavy stress to lower their energy consumption since 2022.
The idea is that if you don't measure, you can't tell where you are wasting resources. So a sensor sounds like a good idea. Except that it's expensive. A current sensor is typically costing a few tens of $, a bit more if you want to have a network able one.
If you want to control a factory, we are talking of tens of sensors, that must be installed, checked regularly, controlled, updated, and supervised.
Installing such a device is not that complex (it takes a couple of minutes), but you have to take care of many aspects:
Is it easy to access once installed, because you may want to change its battery
Is it protected against chocks, water, etc
Has it a decent network access
The other kind of sensors are pretty intrusive. Typically, if you want to measure water consumption, you need a plumber to install it. Costly...
Or you can install such devices on your counter, but it's a global measure, when you want to know how much water is used by a specific machine, or room.
Bottom line, the cost of a single sensor is really minor compared to the cost of managing it in the long run, questioning the ROI.
Here be dragons
So let's say you have decided to monitor your factory. Good.
I hope you have cautiously selected you sensor providers. Because it's a tough ecosystem. What if it goes bankrupt? Have you a stock of spare devices, just in case?
I see it happening: a superb installation, top notch sensors. Except that when the company went down, those fragile sensors that have a 3 years life expectancy where impossible to replace. So the company had to replace *all* if the sensors, paying the price for that, plus the installation, software modification, etc. Pretty much spending the same money than for the initial setup.
What about security? In another use case, the user wanted to leverage pretty old webcams they payed good money for. For sure, they were still working. But the only supported secured protocol was TLS 1.0, which is now unsupported by all the existing browsers. Typically, embedding the video into today's browser is just difficult (proxy required with a TLS downgrade in the middle, etc...)
What about security, again? So you have deployed tens of sensors, and now you realize that non secured transmission of data is a no go. Do you have a way to install certificates, and to manage them properly (expiration, renewal, etc)? Of course access through a strong password is a must. But what about OTP? Does your device support it? Can it be upgraded remotely? Do you have a centralized system to manage all those updates for all your sensors without having to send someone on site to do it one sensor after the other? Not to mention software updates...
Generally speaking, people tend to think that sensors measuring non critical information like electricity consumption does not require a high level of security. Maybe... Until your factory is destroyed like what happened in Iran.
Also keep in mind that many sensors are used in a non secure environment, using antiquated protocols (just giving an example), managed by brittle pieces of software.
Measure is only one part of the equation!
Ok, so you have measures coming in. Now, what are you going to do with it?
If you don't have a software to leverage the flow of data, well, it's useless. And this is where it's start to be complicated.
Your software platform should be able to build decent dashboards, with alarm. It should also be able to manage the sensors status (battery, software updates, sensors security). More important, adding new sensors should be easy. Last, not least, you should be able to navigate through all the data in an efficient way (like, comparing data across the years).
This is where things get *very* complicated. You'll spend many times more on setting up such a platform than what you have spent on devices.
First because you have to define precisely what you want to obtain, which is hard. Second because your requirements will change across time, so you'll have to update your dashboards. Third, your platform should be versatile enough to embed new sensors, new measures, etc. Last, not least, this platform should work for years!
All in all, this is complicated and costly.
What about performances?
So you have picked your platform, fine. It works great, just becoming slower and slower...
Why? Because you have added many sensors. Because you have increased the measure frequency. Because the software is not well written...
Let's say you have 1000 sensors, sending a measure every 5 minutes, 24x7. That means:
200 measures per minute, a bit more than 3 per second. So far, so good.
12 000 measures per hour
288 000 measures per day
Almost 10M per months
100M per year
Now we are talking. If your platform is not properly tuned, when you request a sum up for the year, you bring back millions of data from your database. And generally speaking, your browser is the part in charge of showing the beautiful graph with those millions of dots... Not to mention your database that will be pounded during such a request, and the gigabyte of data crossing the network...
Did I mention the terabytes needed to store years of data? In the previous list, storing 100M entries requires tens of gigabytes (assuming each measure is encapsulated into a 100 bytes piece of data). Sadly, most of the time those pieces of data are stored as JSON (it's so easy to read and process!!!), increasing the original payload size by a factor 2, 3 or more. Add to that you want indexes to speed up direct access. Many indexes...
One sad example: a user who has designed a dashboard that exposes monthly graphs. The sent requests are fetching all the data from many sensors from the 1st day to the current day. Strange enough, the dashboard and the underlying servers are pretty responsive on the very beginning of the month. Not so much when it comes to doing the same request on the 30th...
The problem is that those dashboards are expecting to be easy to develop: "Even a manager can do it!". Short answer: NO.
Last, not least, and I didn't mentioned it, between your data and your dashboards, there is a piece of software intended to manage the workflow: how to process incoming data, convert them properly, doing something useful out of it (like sending alarms, etc). This is the heart of your system, and you generally don't see it... And this is the most complex part!
What about failures?
Ok, you have a working system. Your sensors are setup, their battery are full, they send you back the data without losing the network connection, the workflow and your dashboard have been carefully designed.
Your 1000 sensors sending data every 5 minutes are easily processed on the flow. All in all, 3 messages per second is not that much!
Except that for some unexpected reason you haven't received any data for a full week-end. Hopefully, all the sensors aren't sending the data directly to your system, there is a gateway in the middle that has a beautiful feature: it can gather the data and keep them available until you connect your middle ware.
That means during this week-end, 600K messages have been stored in the gateway, and your admin was able to restore the machine which was running your middle ware on Monday morning (somebody has made a mistake and changed a rule in the firewall, damnit!)
And suddenly your middleware is flooded with half a million messages! Not counting the 200 messages per minutes that are also coming. It will take tens of minutes to process all those data, and it will potentially stress the server to a point they might crash. And the manager who have an urgent need to access the dashboard, because he needs to print a beautiful graph for the 10AM presentation he/she gives to the company board... (yeah, I have seen that. Many times!).
Here, I'm talking about non critical measures, but there are many use cases where we are talking about critical data (think about a factory, or a construction site requiring a real time monitoring).
I have not yet mentioned backups, failover, up time, etc.
Miscellanea
I have faced some quite weird issues during those last 5 years... One of the 'funniest' one was when we installed some sensors into a building. They were not connected with wire, but through a LORA network.
(LoRaWan is a dying radio transmission protocol. It was quite everywhere a few years ago, because it's cheap. Anyway, that's not the issue.)
So we had gateways in the building, a square 4 storey one, with a inner patio. The idea was to install a gateway per floor, in one corner.
Except that it never worked properly. We were losing data for half of the sensors. So we had to send someone with a radio monitoring device to understand why we were losing data...
Bottom line, the windows on the patio were using armored glass with a thin layer of metal to protect against the sun shine... Faraday all over!
Another 'funny' story was the insulated site we were installing, with a 128Kb network access. Barely enough to access it on a text monitor, clearly insufficient to expose 10Mb web pages produced by the dashboards...
Conclusion
I have just scrapped the top of the various issues you will face when conducting such projects. Not to mention that when it comes to energy monitoring, you also have some legal and financial constraints (just because there is a CO2 market, so every ton of CO2 you spare has a value, but you have to *prove* you have really spared it...)
This is quite a rich ecosystem, not so mature, still with a LOT to do.