Outputting DS18B20 temperatures on a LCD1602 – Raspberry Pi Temperature Monitoring Part 4

This is part of a tutorial series. If you feel a bit lost, I suggest following the tutorial in order:

Just interested in the code? – https://github.com/albertherd/DS18B20Reader

This article assumes that you’ve configured one or more DS18B20 sensors to your Raspberry Pi and configured your Raspberry Pi to work with an LCD1602. If you did not, read the above mentioned links.

Okay – this should be a quick post – most of the heavy lifting is done. Remember the LCD1602 library that we’ve used in the second tutorial? We’ll be using that to simply get the temperature we’ve captured in third tutorial and display it.

We’ll need to do the following changes:

  1. Import the LCD1602 library.
  2. Initialize the LCD1602 on application startup.
  3. Read and display information on the LCD1602.
  4. Cleanup resources before exiting – this is important since we’ll need to turn off the backlight after usage.

Import the LCD1602 library

Get a copy of this repository – either by cloning or simply copying lcd1602.c and lcd1602.h files to your solution. We’ll be also adding them to our CMake file – it should look something of the sort. I’ve left the solution on debug in this case.

set(SOURCE main.c sensor.c lcd1602.c main.h sensor.h lcd1602.h)
set(CMAKE_BUILD_TYPE Debug)
add_executable(DS18B20Reader ${SOURCE})

Now it’s simply just adding a reference to the lcd1602.h in the solution. Next!

Initialize the LCD1602 on application startup

We’ll need to initialize the library and open a connection to the display under the right address – check your device address by checking the third tutorial. To simplify things, I hard-coded the value which is 0x27 in my case. Call this when initializing the application.

void InitializeLCD()
{
    int rc;
	rc = lcd1602Init(1, LCDADDRESS);
	if (rc)
	{
		printf("Initialization failed; aborting...\n");
		return;
	}
}

Read and display information on the LCD1602

We’ll be modifying our main loop to output content to the console (not important though and output to the LCD1602 screen. The main adjustment we’ve did in the main loop is that we’ve broken down our output to two functions – Outputting to console (not important) and outputting to the LCD1602 – Let’s see the main loop:

void ReadTemperatureLoop(SensorList *sensorList)
{
    while(!sigintFlag)
    {
        for(int i = 0; i SensorCount; i++)
        {
            float temperature = ReadTemperature(sensorList->Sensors[i]);
            PrintTemperatueToLCD1602(sensorList->Sensors[i], i % LCD1602LINES, temperature);
            LogTemperature(sensorList->Sensors[i], temperature);
        }       
    }
}

Let’s now have a look at the important method – PrintTemperatueToLCD1602. Keeping in mind that the LCD1602 has two lines, we’ll be receiving the calculated line number as a parameter. This will make sure that values will lie between 0 and 1 only using modulus.

We’ll also need to remember that each line will hold up to 16 characters, so we’re truncating anything more than 16 characters (actually 16 + 1 for null termination). We’ll then just pass the (potentially truncated) string to the LCD1602 and et voila!

void PrintTemperatueToLCD1602(Sensor *sensor, int lineToPrintDataOn, float temperature)
{
    char temperatureString[LCD1602CHARACTERS + 1];
    snprintf(temperatureString, LCD1602CHARACTERS + 1, "%s : %.2fC", sensor->SensorName, temperature);

    lcd1602SetCursor(0, lineToPrintDataOn);
    lcd1602WriteString(temperatureString);
}

Cleanup resources before exiting

After we’re done, it’s just a matter of cleaning up resources. As previously mentioned, this is important since we’ll need to turn off the backlight after usage. In the cleanup method, we’re just calling the lcd1602Shutdown method.

void Cleanup(SensorList *sensorList)
{
    printf("Exiting...\n");
    FreeSensors(sensorList);
    lcd1602Shutdown();
}

Fetch a complete copy of the code from from GitHub – https://github.com/albertherd/DS18B20Reader

Let’s run the application! In a terminal with git and cmake installed, run the following commands

git clone https://github.com/albertherd/DS18B20Reader
cd ./DS18B20Reader
cmake . && make && ./DS18B20Reader "Sensor"

With some luck, your LCD1602 should display something like the below. In my case I have two sensors so I’ve fired up the application using the following syntax:

./DS18B20Reader "Sensor1" "Sensor2"

LCD1602OutputCropped

Until the next one!

Using C to monitor temperatures through your DS18B20 thermal sensor – Raspberry Pi Temperature Monitoring Part 3

This is part of a tutorial series. If you feel a bit lost, I suggest following the tutorial in order:

Just interested in the code? – https://github.com/albertherd/DS18B20Reader

Since you made it here, great! Your Raspberry Pi should have one or more DS18B20 thermal sensors connected, like the image below.

sensor2.png

Now that we have our DS18B20 thermal sensor connected to our Raspberry Pi, it’s time to do some programming to read out the temperature! Our application will need to be able to the following tasks:

  1. Discover all the DS18B20 sensors (in my case, I’ve connected 2 but this application should handle an arbitrary number of sensors).
  2. Assign a friendly name so we’ll know which sensor is which and store them in a list.
  3. Retrieve and parse the information from the device.
  4. Do whatever necessary with the gathered information.

1) Discover all the DS18B20 sensors

First and foremost, our code cannot just assume that devices just exist on the system – we’ll need to go and discover these devices. Since the DS18B20 makes use of the 1-Wire protocol, devices will live under the /sys/bus/w1/devices/ directory. Therefore, our code will need to devices living under this directory, whose names start with 28 since all DS18B20 device names will start with 28. Let’s start by knowing how many devices are connected.

typedef struct Sensor
{
    char *SensorName;
    FILE *SensorFile;    
} Sensor;
 
typedef struct SensorList
{
    Sensor **Sensors;
    int SensorCount;
} SensorList;

DIR *dir;
struct dirent *dirEntry;

SensorList *sensorList = malloc(sizeof(SensorList));
sensorList->SensorCount = 0;

if(!(dir = opendir("/sys/bus/w1/devices/")))
    return sensorList;

while((dirEntry = readdir(dir)))
{        
    if(strncmp(dirEntry->d_name, "28", 2) == 0)
    {
        sensorList->SensorCount++;
    }
}

2) Assign a friendly name so we’ll know which sensor is which and store them in a list

Now that we’ve discovered the devices connected to the system, it’s time to save a reference and optionally a friendly name as well. Logic mostly applies from step 1.

sensorList->Sensors = malloc(sizeof(Sensor*) * sensorList->SensorCount);
Sensor **currentSensor = sensorList->Sensors;   

int sensorNamesAllocated = 0;
while((dirEntry = readdir(dir)))
{        
    if(strncmp(dirEntry->d_name, "28", 2) == 0)
    {   
        char *sensorName;
        if(sensorNamesCount > sensorNamesAllocated)
        {
            sensorName = strdup(*sensorNames);
            sensorNames++;
        }
        else
        {
            sensorName = strdup("Sensor");
        }

        char sensorFilePath[64];     
        sprintf(sensorFilePath, "%s%s%s",  "/sys/bus/w1/devices/", dirEntry->d_name, "/w1_slave");
        *currentSensor = GetSensor(sensorFilePath, sensorName);
        currentSensor++;
    }
}

Sensor *GetSensor(char *sensorId, char *sensorName)
{
    Sensor *sensor = malloc(sizeof(Sensor));
    sensor->SensorFile = fopen(sensorId, "r");
    sensor->SensorName = sensorName;
    return sensor;
}    

3) Read the temperature from the device

This is the most exciting part – we actually get to read the temperatures! Using the sensor information we got from steps 1 and 2, we can get the device, open it as a file, extract the readings and parse it accordingly. Using the FILE API makes it very easy to do so – grab all the contents and store it in a buffer. As mentioned in the first tutorial, the content of the file looks as follows –

0b 01 55 05 7f 7e 81 66 bf : crc=bf YES
0b 01 55 05 7f 7e 81 66 bf t=16687

We’re only intereested in the t= component, so some string manipulation and float conversion will take the 16687 and convert it into 16.687C. We’re also doing some range checking since the DS18B20 is rated between -55C and +125C

   
long deviceFileSize;
char *buffer;

FILE *deviceFile = sensor->SensorFile;
fseek(deviceFile, 0, SEEK_END);
deviceFileSize = ftell(deviceFile);
fseek(deviceFile, 0, SEEK_SET);

buffer = calloc(deviceFileSize, sizeof(char));

fread(buffer, sizeof(char), deviceFileSize, deviceFile);
char *temperatureComponent = strstr(buffer, "t=");
if(!temperatureComponent)
{
    free(buffer);
    return -1;
}

temperatureComponent +=2; //move pointer 2 spaces to compensate for t=

float temperatureFloat = atof(temperatureComponent);
temperatureFloat = temperatureFloat / 1000;

if(temperatureFloat  125)
    temperatureFloat = 125;    

free(buffer);
return temperatureFloat;

4) Do whatever necessary with the gathered information

We now have the information at hand, great! We can do many sort of things with it, such as sending an email, activating some other device or whatever is necessary. For demo purposes, we’re simply going to output the contents to the console just to see it working. This is in a loop so we’ll keep reading the temperature until the application exits.

 while(1)
    {
        for(int i = 0; i SensorCount; i++)
        {
            char dateTimeStringBuffer[32];
            strftime(dateTimeStringBuffer, 32, "%Y-%m-%d %H:%M:%S", localtime(¤tTime));

            float temperature = ReadTemperature(sensorList->Sensors[i]);
            printf("%s - %s - %.2fC\n", dateTimeStringBuffer, sensorList->Sensors[i]->SensorName, temperature);
        }       
    }

To try out the code as a whole solution, grab a copy from GitHub – https://github.com/albertherd/DS18B20Reader

In a terminal with git and cmake installed, run the following commands

git clone https://github.com/albertherd/DS18B20Reader
cd ./DS18B20Reader
cmake . && make && ./DS18B20Reader "Sensor"

If the output looks like the below, congratulations!

ds18b20 tutorial sample output.png

In the next tutorial, we’ll pick up from here and we’ll start outputting the content on an LCD1602! Until the next one.

Connecting a LCD1602 with an I2C module to your Raspberry Pi – Raspberry Pi Temperature Monitoring Part 2

The LCD1602 is a very famous LCD that can be connected to various devices such as the Raspberry Pi. The LCD1602 on its own is quite tricky to wire it up since it requires 16 pins to be connected. The LCD1602 can also be purchased with an I2C module, which reduces the amount of pins needed to just 4.

For this tutorial, we’ll be working with a LCD1602 with an I2C module. I got mine from AliExpress for around $2.50. Make sure to grab a set of jumper cables as you’ll need them to connect the LCD to the Raspberry Pi. I got mine from AliExpress as well for around $1.50.

IMG_20190102_110607.jpg

img_20190109_230900

Let’s start by wiring it up. We have 4 pins connect – GND (ground), VCC (power, 5V), SDA (data line) and SCL (clock line). GND and VCC can be connected to any equivalent GND and 5V pin. SDA and SCL should be connected to pins BCM 2 and BCM 3 accordingly.

lcd1602_i2c_raspberrypi

img_20190109_231321

If you’re following the Raspberry Pi Temperature Monitoring Part 1 and connected the DS18B20 temperature sensors, you should now have the following configuration.

lcd1602_i2c_ds18b20_raspberrypi

img_20190109_232649

Great! We’re done from the hardware’s side – let’s start configuring our Raspberry Pi to communicate with our LCD.

Firstly, let’s enable I2C from the Raspberry Pi Config. Fire up the raspi-config to get started: sudo raspi config

Now navigate to Interfacing Options => I2C => Enable I2C

raspi-config-interfacing-options

raspi-config-interfacing-options-i2c

Now that we’ve enabled I2C communication, it’s time to start development! We’ll need to get some tools before we start working though, so fire up a shell and input:

sudo apt-get install i2c-tools.

Once that’s done, the LCD is ready to be programmed! Let’s make sure that the LCD is properly connected and working. In a shell, type:

i2cdetect -y 1.

The output should be something like the below. Note the number outputted by the command; will be needed later on. We’ll need this address when we’re trying our demo code. In this case, the address is “27”.

i2cdetect

Great! Now, it’s time to test out our display and see if it works! We’ll be using a Github library – https://github.com/albertherd/LCD1602. This has been forked from https://github.com/bitbank2/LCD1602. We’ll be using my fork since the original repository has an unresolved issue with clearing the display.

After you’ve cloned the repository in your working directory, it’s time to use the address (27 in my case) obtained earlier. Open the main.c and find the call to lcd1602Init and change second parameter. This is how it looks in my case:

lcd1602Init(1, 0x27);

Now it’s time to compile and run our code. If all goes well, we should be getting some text on the screen. You can change the text to whatever you’d like by changing the following lines in main.c.

lcd1602WriteString("BitBank LCD1602");
lcd1602SetCursor(0,1);
lcd1602WriteString("ENTER to quit");

Build and run using the following commands:

make
make -f make_demo
./demo

The screen should look like the below:

IMG_20190110_231937.jpg

Great! Now we’ve successfully connected our LCD1602 to our Raspberry Pi and we’re able to output content on it!

In the next part of this tutorial series, we’ll start by capturing the temperature using the sensor in our first part of the tutorial and outputting it! Stay tuned.

Connecting a DS18B20 thermal sensor to your Raspberry Pi – Raspberry Pi Temperature Monitoring Part 1

A project that I’ve been working on during the Christmas holidays was to hook up some thermal probes to my Raspberry Pi, just to play around. This tutorial simply follows the steps that I’ve taken to achieve so.

You’ll need:

  • Raspberry Pi, any flavor as long as it has GPIO headers available. I had a Raspberry Pi 2, so I used that.
  • You’ll also need the usual suspects – USB to MicroUSB to hook it up to power, HDMI to connect it to a display for initial configuration and an ethernet port to manage it through SSH. I highly recommend configuring SSH rather than using the device itself. This tutorial assumes you’re using SSH.
  • A DS18B20 sensor – I’d suggest getting one which includes a Plugable Terminal to avoid soldering – just wire it up and you’re good to go. I got mine from AliExpress
  • Also make sure your kit has 3 jumper cables. They are typically included. Just to be sure, I also got a set of female to female jumper cables from AliExpress though I did not use them for the DS18B20 sensor.

All right, let’s wire it up! The DS18B20 sensor requires three pins – data, VCC (3.3V), and ground. Connect the wires as below. Data is yellow, VCC is red and ground is black.

IMG_20190102_105252

Connect the 3 pins using the jumper cables as shown below.
sensor1.png

IMG_20190102_105726

We’ll also need to instruct the Raspberry Pi that we’re going to connect the DS18B20 sensor. This sensor makes use of the 1-Wire protocol, so let’s activate it:

  • Connect to the Raspberry Pi using SSH
  • Let’s start by editing the config file that the Raspberry PI parses every time it boots up: sudo nano /boot/config.txt
  • Go to the end of the document and input the following. Specifying gpiopin=4 is actually optional since by convention, 1-wire devices are expected on gpiopin 4 on the Raspberry Pi.
    # Enable OneWire Protocol
    dtoverlay=w1-gpio;gpiopin=4
  • Time to reboot the Raspberry Pi sudo reboot
  • Once the Raspberry PI reboots and you re-connect using SSH, it’s time to get data from the sensor! Let’s find the 1-wire devices connected to the system. Let’s start by browsing to the appropriate directory. cd /sys/bus/w1/devices
  • Great! Let’s now see the devices attached to the Raspberry Pi. ls
  • This will get the devices attached using the 1-Wire protocol. You should have a device called 28-xxxxxxxxxxxx (where x stands for your unique 12 digit serial number). Let’s now browse the device. Mine is 28-02199245e07b, so let’s use it an example. cd 28-02199245e07b
  • Once you access the device, there should be a file called w1_slave. Let’s see the contents of the file. cat w1_slave
  • The file should look like this:
    0b 01 55 05 7f 7e 81 66 bf : crc=bf YES
    0b 01 55 05 7f 7e 81 66 bf t=16687
  • If the file looks like the above, great! The temperature component is t=16687. The temperature in this case is 16.687 °C

We also can take this to the next level and add another thermal probe! Attach it as shown below.
sensor2IMG_20190102_110134

This will require re-editing the /boot/config.txt. Let’s do it!

  • Re-open /boot/config.txt – sudo nano /boot/config.txt
  • Go to the end and add the following. I chose pin 24 because it’s easy to wire since it’s close to a 3.3v and ground. dtoverlay=w1-gpio;gpiopin=24
  • Close and save, then cd /sys/bus/w1/devices
  • You should now see two devices as 28-xxxxxxxxxxxx

Of course, at this stage we did get the temperature, but it’s not really usable. We can get access to this information programmatically – this is what we’ll be doing in the next part of this tutorial. We’ll also be eventually showing the information on a separate LCD screen! Stay tuned!

Group files into folder structure by date using PowerShell

Lately I was sorting out around 5000 files worth of pictures and videos taken out from a mobile phone, to upload them on a cloud drive. To make this task more manageable, I wanted to separate these files out, firstly by file type and then by date. This will make the folders far more smaller and therefore easier to upload these files to the cloud folder by folder.

Grouping such several files manually will be very tedious and error-prone. This task is a perfect candidate to be scripted out; this is exactly what I did. Since this may be helpful to other people going through such task, I decided to share this script.

All the script requires is a source path and a target path – anything else is handled by the script. It has several assumptions, such as it always moves, never deletes a file. Customising the script is easy and only requires basic programming knowledge.

Check it out here or below: https://github.com/albertherd/GroupFilesPowershell/blob/master/GroupFiles.ps1

Param(
    [Parameter(Mandatory=$true)]
    [string]$sourcePath,
    [Parameter(Mandatory=$true)]
    [string]$targetPath
)

$global:fileTypeLookup = @{};
$folderDateTimeFormat = "MM-yyyy"

function Copy-FilesIntoFoldersByMonthAndType{
    param()

    $files = Get-ChildItem -Recurse -File -Path $sourcePath
    $filesProcessed = 0

    foreach($file in $files){
        $folder = Get-DirectoryForFile $file
        Copy-Item $file.FullName -Destination $folder

        $filesProcessed++
        Write-Progress -Activity "Grouping files" -Status "$($filesProcessed) out of $($files.Count) grouped" -PercentComplete (($filesProcessed / $files.Count) * 100)
    }
}
function Get-DirectoryForFile{
    param($file)

    $monthYearDirLookup = Get-FilePathDictionary $file
    $modifiedTimeMonthYearInternal = $file.LastWriteTime.ToString("MMyyyy")

    if($monthYearDirLookup.ContainsKey($modifiedTimeMonthYearInternal)){
        return $monthYearDirLookup[$modifiedTimeMonthYearInternal]
    }

    $extensionWithoutDot = $file.Extension.Substring(1, $file.Extension.Length - 1)
    $dateFolderFileName = $file.LastWriteTime.ToString($folderDateTimeFormat)
    $newPath = $targetPath + "\" + $extensionWithoutDot + "\" + $dateFolderFileName
    $path = New-Item -ItemType Directory $newPath -Force

    $monthYearDirLookup[$modifiedTimeMonthYearInternal] = $path.FullName
    return $path.FullName
}

function Get-FilePathDictionary{
    param($file)

    if($global:fileTypeLookup.ContainsKey($file.Extension)){
        return $global:fileTypeLookup[$file.Extension]
    }

    $global:fileTypeLookup.Add($file.Extension, @{})
    return $global:fileTypeLookup[$file.Extension]
}

Copy-FilesIntoFoldersByMonthAndType

In my case, it generated the below folder structure:

FolderStucture

Far more manageable than one flat folder of 38.8 GB folder, for sure!

Until the next one.

GDPR and the Maltese Political Propaganda

Unless you’ve been living under a rock, GDPR is now in effect as of 25th May 2018. I want to discuss the impact of GDPR, with regards to the Maltese Political Propaganda. This new law brings up a lot of changes and rights to the end user – the below is a summary which should be easily understandable without any jargon.

Politicians CAN send you personalized letters through postal services

Politicians do have the right to obtain your home address through three main channels: Electoral Register, Online Directory and Printed Directory. Though, these letters must have clear instructions on how one can easily opt out of these personalized letters and they must provide a method to do so, such as a mailing address, email or contact number.

Politicians CAN call you on your telephone / mobiles

Politicians do have the right to obtain your telephone / mobile again through three main channels: Electoral Register, Online Directory and Printed Directory. Though, the caller must respect the fact that you can ask them to hang up and remove your personal details so that they do not contact you again. One can also opt out of the directories in order to hide their telephone / mobile number, through their phone operator.

Politicians CAN contact you on election day to encourage you to go out and vote

Politicians can contact you using different mediums, which may be obtained from the following: Online Directory, Printed Directory public Social Media information or any publicly made information that the end user has made public himself. Of course, you can ask whoever is contact you to stop any communications and immediately delete your personal information.

Politicians CANNOT send you e-mails UNLESS you have given them an explicit consent and your email directly

In order for a politician to obtain your e-mail address, you must give it to them. This is typically done through direct political channels, such as propaganda websites. Also, these channels must provide an easy way to opt out of these emails, such as a unsubscribe link, which is clearly labelled and visible appopriatelty. The unsubscribe link must be send with each and every email.

Politicians CANNOT send you SMSs UNLESS you have given them an explicit consent and your phone number directly

In order for a politician to obtain your phone number, you must give it to them, exactly like the previous case with e-mails. With each SMS that the subscribed user receives, the sender must provide a contact number or email with each and every SMS sent, which the receiver can use in order to opt out of such SMSs.

3rd Party contractors which may send Propaganda SMSs (such as Go / Vodafone) MUST ensure that the politican has obtained the proper consent of the end user

If the politician does not provide clear and legitamate proof of consent to the 3rd Party contractor, the contractor must refuse to send propaganda SMSs to end users.

Liking / Following a politicians’ account on Social Media means that you’re giving him consent in order to obtain your PUBLIC Social Media information, such as e-mail or phone number.

Any information that you publicly post on Social Media can be accessed and used for political propaganda, by simply liking / following a page / account.

Some additional information:

  1. You ALWAYS have the right to opt-out of any propaganda messages, whether’s it’s postal, email, SMS or any other means, even if it’s something that has not been invented yet.
  2. Consent is ALWAYS needed for electronical communications, whether it’s email, or some futuristic electronic holographic message, consent is ALWAYS needed.
  3. Personalized letters can be sent through any means, be it postal services, or some fancy pigeon delivery. Keep in mind that you can always opt out of these letters, but by default, you’re in.
  4. If you’re tired of receving door-to-door political propaganda, you can stick a “no junk mail” message with your letterbox, and they must respect this fact.
  5. Politicians CANNOT charge you money to opt you out of any communication

Note that I’ve omitted some less important information in this article, such as dealing with Automated Calling Machines. All this information has been obtained from the Office of the Information and Data Protection Commissioner. If you feel that your rights have been violated, you can submit a report here.
Note by all means this article is not a legal advice, merely a guideline on your rights.

Source of article is here

Google Hash Code 2018 solution and source code – 1st in Malta and top 20% worldwide

Our solution: https://github.com/albertherd/Hash-Code-2018-Team-Stark

I recently had the honour to team up with my friends (and co-workers) to have a crack at the Google Hash Code 2018 online qualification round. We went to compete with little to no expectation. In fact, by the first two hours of the competition, we had almost nothing!

Google Hash Code is a team based competition, where teams all over the world gather together to participate and solve a Google engineering problem, using a language of their choice.

The problem was: Given a set of rides and a number of cars in a city grid, assign rides to cars, and serve the rides in a restricted time. There are also bonuses if you manage to start the ride as early as possible. Here is the Problem statement, input data set A,  input data set B input data set C  input data set D,  input data set E,

Our solution is not the cleanest solution, I must admit. One has to keep in mind that when undertaking these competitions, the biggest challenge is to come up with a decent solution the problem in a very short amount of time, rather than creating a complex solution which does solves the problem perfectly.

However imperfect, hacks-infested and dirty, it managed to place us first in the Maltese islands, and in the top 20% worldwide (more specifically, 895th).

Capture

Our solution does not make use of a Genetic Algorithm; we tried to create a Greedy Algorithm with several optimizations in order to boost the quality of the output. Optimizations include:

  • Having the main loop revolve around steps, rather than cars or rides (much like a game engine main loop).
  • Sort all the initial rides by start time.
  • Discard any rides that have been already undertaken.
  • When trying to pick out a ride, we make sure that given current time (or step), we check if the ride can actually be served on time. Using this thought, we’ve managed to produce output with 0 late rides across the board!
  • Get the first feasible N rides – sort them out by closest, then by travelling and undertake that ride. Since each ride yielded the same bonus, irrespective of the distance traveled, the shortest rides were always the most profitable.
  • Other minor optimizations.

There are multiple optimizations and enhancements that we wished to include, such as removing those impossible rides from the data set as we go along, and distributing each car / ride according to a “zone”. In fact, the code does have a Zone class, which never got used.

Have a look at our solution : https://github.com/albertherd/Hash-Code-2018-Team-Stark

Until the next one!

Performance differences when using AVX instructions

Download source code from here

Recent news on exploits on both Meltdown and Spectre got me thinking and researching a bit more in depth on Assembly. I ended up reading on the differences and performance gains when using SIMD instructions versus naive implementations. Let’s briefly discuss what SIMD is.

SIMD (Single instruction, multiple data) is the process of piping vector data through a single instruction, effectively speeding up the calculations significantly. Given that SIMD instruction can process larger amount of data in parallel atomically, SIMD does provide a significant performance boost when used. Real-Life applications of SIMD are various, ranging from image processing, audio processing and graphics generation.

Let’s investigate the real performance gains when using SIMD instructions – in this case we’ll be using AVX (Advanced Vector Extensions), which provides newer SIMD instructions. We’ll be using several SIMD instructions, such as VADDPS. VSUBPS, VMULPS, and VDIVPS. Each instruction is responsible for adding, subtracting, multiplying and dividing single precision numbers (floats).

In reality, we will not be writing any Assembly at all, we’ll be using Intrinsics, which ship directly with any decent C/C++ compiler. For our example, we’ll be using MSVC compiler, but any decent compiler will do. The Intel Intrinsics Guide provides a very good platform to look up any required intrinsic functions one may need, thus removing the need to write Assembly, just C code.

There are two benchmarks for each arithmetic operation: one is done naively and one is done using intrinsics thus using the necessary AVX instruction. Each operation is  performed 200,000,000 times thus to make sure that there is enough time to demonstrate it for a benchmark,

Here’s an example of how the multiplication is implemented naively:

void DoNaiveMultiplication(int iterations)
{
    float z[8];

    for (int i = 0; i < iterations; i++)
    {
        z[0] = x[0] * y[0];
        z[1] = x[1] * y[1];
        z[2] = x[2] * y[2];
        z[3] = x[3] * y[3];
        z[4] = x[4] * y[4];
        z[5] = x[5] * y[5];
        z[6] = x[6] * y[6];
        z[7] = x[7] * y[7];
    }
}

Here's an example of how the multiplication is implemented in AVX:

void DoAvxMultiplication(int iterations)
{
__m256 x256 = _mm256_loadu_ps((__m256*)x);
__m256 y256 = _mm256_loadu_ps((__m256*)y);
__m256 result;

for (int i = 0; i < iterations; i++)
{
result = _mm256_mul_ps(x256, y256);
}
}

Finally, let's take a look on how the results look:

naivevsavx

 

AVXPerformanceGains
Performance gains when using AVX

From the graph above, one can see that when optimizing from naive to AVX, there are the following gains:

  • Addition: 217% faster – from 1141ms to 359ms
  • Subtraction: 209% faster – from 1110ms to 359ms
  • Multiplication: 221% faster- from 1156ms to 360ms
  • Division: 300% faster – from 2687ms to 672ms

Of course, the benchmarks show the best case scenarios; so real-life mileage may vary. These benchmarks can be downloaded and tested out from here. Kindly note that you’ll need either an Intel CPU from 2011 onwards (Sandy Bridge), or an AMD processor from 2011 onwards (Bulldozer)  in order to be able to run the benchmarks.

 

 

UPDATED: Intel and its flawed Kernel Memory Management Security

It has emerged that Intel CPUs made in the last decade or so are missing proper checks when it comes to securing Kernel Memory. It would seem that through special (undocumented) steps, a User-Mode application can peek and make changes to Kernel-Mode Memory. This means that any application, such as your browser, can access and change your system memory.

Some theory

In the 32-bit era, an application could typically access up to 4GB of RAM; this has been de-facto for ages. What really happened is that the application had access to 2GB of for User-Mode memory (used to typically hold the memory needed by the application to function). The other 2GB is mapped to Kernel space, containing memory locations for Kernel-Mode memory.

In the 64-bit era, these memory limitations were lifted since a 64-bit architecture can access such a larger address space (16 exabytes, to be exact). Given that the Kernel-Mode memory is so much larger (248TB), the OS can randomly place it anywhere it pleases, randomly. This randomness (Address space layout randomization) successfully makes it so much harder for foul-playing applications to find the addresses of Kernel-Mode functions.

So, what’s happening?

Typically the code that runs in User-Mode (typical code) does not have access to the Kernel-Mode memory. The reason why this is done is so when an application switches to Kernel-Mode (needed for example to open a file from disk), the Kernel-Mode memory would still be accessible, avoiding the needed to have 2 memory tables, one for User-Mode and one for Kernel-Mode. Having more than one table will mean that during every sysenter (or equivalent), tables will need to be swapped, cache needs to be freed and any overhead that such operations require.

It would seem that on Intel CPUs, hackers have found a way to bypass this security feature. This means that a User-Mode application can now access Kernel-Mode memory; which is devastating. A User-Mode application can apply small changes to the Kernel and change its functionality. Since an application has access to Kernel memory, a hacker can basically do whatever he pleases with the target’s system.

How can this be fixed?

Unfortunately, an easy fix is not available. The whole memory management logic needs to be re-written, so that instead of having just one memory table, which maps both User-Mode and Kernel-Mode memory, an additional table will hold the Kernel-Mode memory; this table will be only accessible from Kernel-Mode memory. The change is being dubbed as Kernel page-table isolation (KPTI, known as KAISER).

Adding a new memory table and switching to-and-fro has negative effects on the overall system performance, especially in I/O heavy applications. The reason is that I/O involves a lot of User-Mode and Kernel-Mode switching. Given that the new code needs to run every time the system switches from User-Mode to Kernel-Mode. performance degradation are expected. Unofficial figures quote between 5%-30% performance impact, depending on the application. OC3D has provided some benchmarks; FS-Mark (I/O benchmark) show a devastating hit in performance. PostgreSQL said that there is a best case of 17% slowdown, worst case of 23% using new new Linux patch.

Which operating systems are vulnerable?

Basically, all Operating systems are vulnerable to this hack. this is because this is a bug that goes beyond the operating system, since it lives on the CPU rather than on an operating system level. Scary! Vendors have been (secretly) informed of this issue and are working on fixing the vulnerability:

Are non-Intel CPUs vulnerable?

All we know at the moment is that AMD CPUs are NOT vulnerable. This has been confirmed by AMD themselves. In fact, Tom Lendacky from AMD has issued a fix for the Linux kernel itself, adding a check so that if the CPU is AMD, the mitigation is not applied.

What’s next? How can I stay safe?

If you got an AMD CPU, well then congratulations, you’re safe! If you’re on an Intel System, don’t panic just yet. Yes, you are vulnerable, but yes, you still control what you do with your computer. If you don’t visit dodgy websites and don’t install dodgy applications, you’ll remain safe. But that’s age-old advice.

 

 

BOV, you’re a good bank, but your app SUCKS!

As Christmas dawns on us, most of us go out to do the usual Christmas shopping. As of this year, I’ve decided to ditch using cash (where possible) and only use bank cards when shopping. Sounds great right? Well, almost.

I am a BOV customer and I got two main accounts with them: a Cashlink account (where my wage is deposited) and another account, which has a VISA account bound to it. Normally, I only put the money that I’ll be spending on this account, as a safety measure. This means that every time that I need to do a purchase, I use the BOV app to do the transfer. This is where it quickly goes south.

So, last time I was at Valletta doing some shopping, I went to transfer some money to my VISA account. I fired up my BOV app, logged in and bam, “Application Error”. Let’s try again – but now, when I logged in, I got disconnected from my Wi-Fi and got connected on 4G. Great, the connection dropped again! Third time’s the charm! I re-logged in, and was greeted by the beautiful “you are already logged in” message. Needless to say, I had to wait again, switch off my Wi-Fi to make sure I don’t connect to somewhere else and then actually manage to to the transfer. Far more painful than it should be!

Anyway, this was not the first time that I’ve had this issue. Actually I’m surprised when it works the first time round! I’ve been talking about this with my friends on how painful it is and one of my friends introduced me to Revolut. Basically, it’s an online bank which provides you with a very good mobile app and a MasterCard. I’ve ended up replacing my BOV VISA with the MasterCard from Revolut, and I regret nothing. There are multiple advantages to using a alternative bank, but for the scope of this blog, it’s a bit irrelevant.

So, what can BOV do to their app to win me back? Because, in reality I’d still rather do all my banking with BOV rather than an alternative bank, but BOV has so much work to do beforehand.

1) I don’t care if I’m logged in from another location

Why is this a feature in the first place? If I want to manage my money from my BOV app and BOV Internet Banking, so be it! Anyway, this issue happens because you’re disconnected from the BOV app before hitting logout. This is the worst issue of them all if I’m honest.

2) The app is slow

It seems that the app is always sluggish. Why does it take 5-10 seconds to log in and get my balance? Coming to think of it, every screen transition takes about 5-10 seconds! Hey, at least they fixed the issue of waiting a minute on “Checking Security” popup. That wasn’t fun!

3) The UI is STUPID

I got a full HD (1080*1920) screen, but it seems that the BOV app can only use 7% of the screen to place the username and key fields. Why are they so tiny? Even worse, it seems that allowing multiple users to log in from the same app is a bit daft in the first place; it should only ask for my password (and remember my user).

4) The UI is dated

I get the feeling that this application was designed when Android was still in version 2 or 3. The UI is very dated – the way that settings appear reminds me of old Android. By the way, why is the settings tab activated by a tiny hamburger icon, when clearly there’s a LOT of space available?

5) Why am I not allowed to make payments to any IBAN I desire?

I can make payments to my friends (if I got their mobile number), some list of hardcoded shops (I assume vetted by then), top up my phone and that’s it! If I need to make a payment to some IBAN, I cannot do it through the app; I’ll need to use their online portal.

6) Where is fingerprint authentication?

I assume the answer is “they can’t be bothered”. Obviously, since they haven’t done any decent update to the app since like forever, this feature is stuck in oblivion. Before I got my Revolut account, I never appreciated the comfort of logging in using your fingerprint, and believe me, it works GREAT.

7) I’d like more fine-grained security control my cards

We live in 2017, but it seems that BOV is living in 1017. Why I am not allowed to turn my VISA on and off on the fly? While you’re at it, I’d like fine-graned control on swipe payments, ATM and online payments please.

8) Where are the contactless cards?

This is a good one as well, but not related to the app per-se. It seems that although BOV has rolled out contactless cards, I haven’t got one, why’s that? I’ll just assume that I need to go through some hoops and whatnot to get my hands on one. Contactless is AWESOME by the way.

9) Why do pre-authorization payments take days to appear?

I’ve always wondered this. Sometimes payments are pre-authorized rather than an instant withdrawal. When these payments occur, I can only notice this because my book balance is different (less) then my available balance. They’re not written in the statement, for some weird reason. My Revolut account does this and it also notes that this transaction is awaiting confirmation or reversal.

10) Some other features that I’d like

  • A filterable statement, or at least sorted by month
  • Split Bill
  • Request money
  • Freeze card / account

I’m pretty sure that there are a million other things I can nitpick on, but that’s all for now. Until the next one