Low Tech Graph for Low Tech Needs

Following from where I left off in my data collection phase, I need a way to view my data while on my mac. I usually never have my RPi connected to a display, but rather ssh in. I was thinking about making a live update graph from a javascript library and hosting it on the RPi, but, then I realized that I was only collecting 1 datapoint every 6 minutes (a built in characteristic of Transmission) and thought that it would be overkill, especially since I would only want to see the graph while I was on the same local network.

I settled on a module that is a part of scitools, called aplotter, which makes an ascii (not an image) plot, because it would be able to make a plot straight in the same terminal session, and because my demands are more qualitative when it comes to the graph.

My Script (a link to the one I use here) got a few things done:

  • Used regex to parse the strings I had saved in a csv and extracted time and byte strings
    • CSV specific:
      • with open(data_input_path, 'rb') as csvfile:
        • raw_torrenting_data = csv.reader(csvfile, delimiter=',')
          • then you have each line to work with, and then element within that line
    • straightforward: re.search(pattern, source) , then use groups to extract elements
    • would sometimes get a method NoneType error when the parse failed, sometimes for questionable reasons and sporadically, so I introduced a condition the avoid these datapoint errors
    • if str(type(parse_data_point)) == "<type '_sre.SRE_Match'>"
  • turn string times into time objects
    • datetime.datetime.strptime(string_a, '%Y-%m-%d %X')
    • can subtract time objects no problem
      • but working with the deltas is interesting, it only stores days, and then leftover seconds, so you can query days and seconds only
      • hour_difference = (4 + diff_obj.days * 24 + (diff_obj.seconds/3600.))
  • Did the derivative of the cumulative count of uploaded bytes
    • shifted one list, and subtracted their elements
    • slice notation was interesting
      • original_data = array_of_string_nums[1:]
        • leaves off the first element, 0th index
      • offset_data = array_of_string_nums[:-1]
        • leaves off the last element
  • Made converted bytes more readable
  • Made the graph for the last day
    • x = x_axis[-145:-1]
    • y = y_axis[-145:-1]
    • scitools.aplotter.plot(x, y)

What it looks like:

Screen Shot 2013-06-12 at 8.48.19 PM

Good enough! Can see whether uploading is going well, or not at all. Notice that all the time is spent loading the library for the plotter!

  • I found this out by inserting: import time ; start_time = time.time() at the beginning, and then print time.time() - start_time, "seconds after whatever" intermittently

Things to fix later:

  • see if its possible to use another library thats quicker, or some workaround
  • parse command line arguments to plot more than just the last 24 hours
  • consider prettier graphs like the python port of chartkick

Continuous Data Collection (Torrent)

My Ultimate goal: see a graph of my upload speed from Transmission over a long period of time

First goal: dump the upload speed into a file.

I use Transmission as my torrent client on the RPi, and I found that Transmission will save the upload count in a stats.json  file somewhere, to find where, I navigated to my root directory on the sdcard, and did a find . -name "stats.json" (the period means that we start searching from the current directory). I saw what I was looking for: find: `./var/lib/transmission-daemon/info': Permission denied and so I switched to super user via sudo -s and navigated to that directory, and just to double check, did: grep 'upload' stats.json and indeed saw the line: "uploaded-bytes": 186012214246. So indeed, this would be where I would collect my data.

Now, make a file in your favorite text editor, or: nano my script.py at the shell/terminal

Here’s a link to the code I use on my machine, and a breakdown below:

Python essentials:

  • reading a file
    • input_path = '/var/lib/transmission-daemon/info/stats.json'
    • infile = open(input_path, "r")
    • contents = infile.read()
  • appending a file (doesn’t have to exist, but will make sure data is added, and doesn’t replace)
    • useful to make a method, named say: write_data, and then call that method, so put the following in a method that takes one new_datapoint argument
    • output_path = '/media/USBHDD/PythonStudy/torrent_data_collection/data_one.txt'
      • I’ve put this on an external disk, avoid writing to your SD card too many times, it will stop working in no time
    • outfile = open(output_path, 'a')
    • outfile.write(new_datapoint)
    • outfile.close()
  • Regular expressions are very useful, a good place to start once you have some time is from Google’s own python class
    • downloaded_bytes = re.search('"downloaded-bytes":\s(\d+,)', contents)
      • don’t forget to: import re
      • test regular expressions here, yes the / /  instead of ' ' syntax is a Ruby specific thing, but everything else is the same
    • note the inner parentheses, downloaded_bytes is a “match” object, and it has the attribute of group(1) that will return the first string that we surrounded in parentheses
  • Time Stamps
    • current_time = strftime("%Y-%m-%d %X", gmtime())
      • don’t forget to: from time import gmtime, strftime, sleep
  • I added a condition to check if the previous datapoint is equal to last one, and not to call the write_data method if that is the case
    • if current_data != previous_data:
      • indentation is important in Python!
      • write_data(current_time + "$" + downloaded_bytes.group(1))
        • using a dollar sign as separation for reference later
      • previous_data = downloaded_bytes.group(1)
    • then tell it to wait for a while:
      • sleep(5)
  • The actual call write_data method, and the reading of the file,  is wrapped in a loop that will go forever
    • I just did:
      • forever = 0
      • while forever < 1:
        • # the above code…
  • Last but not least, add a: #!/usr/bin/python line at the top
    • or, check where Python is installed on your RPi by doing: which Python in the terminal/shell, and use that directory

Running it

  • Remotely
    • ssh pi@<RPi's local ip address> 'nohup sh -c "( ( sudo python /media/USBHDD/PythonStudy/torrent_data_collection/collection_run.py &>/dev/null ) & )"'
    • and then press Ctrl+Z and then:  bg
  • Just run with sudo on the RPi itself, and press Ctrl+Z, and then: bg to let it keep running
  • Quitting it
    • I use a utility called htop (sudo apt-get htop) from which, if I launch it with sudo htop I can view all my processes, press F4 to filter (by “python”) in this case, and then F9 to kill the process, selection SIGTERM
  • Costs: O% CPU and ~1% Memory, little disk space
    • insignificant considering Transmission takes up anywhere between 40-50%
    • check with the du -h command how big the current directory is
  • live-update your shell with the file’s contents via: tail -f filename.txt, Ctrl+C to exit out

For my reference: placed the script in: /media/USBHDD/PythonStudy/torrent_data_collection , which is the same place the datafile is being made

Next time… making graphs with the data!

OSX’s Disk Utility Analogue for Linux: Gparted

Gparted is a nice graphical disk utility application for linux, its useful for keeping track of partitions (easier than the tail -f /var/log/messages command, though it doesn’t dynamically update on my raspbian install), and changing their format.

I used Gparted to partition and format an HFS+ SSD on linux on the RPi. You can get Gparted via sudo apt-get install gparted. After this installed, and since I already had the hfsplus plugin on my RPi from earlier, I did sudo gparted in the command line after already being logged into the GUI on the RPi (via startx at the startup command line). From there I was able to navigate to the right options to clear my drive, first by going to GParted>Device in the menu and selecting my device, and then Device>Create Partition table… , and then finally by right clicking in the me that popped up below and selecting Apply.

I had to do all this today because of a problem that I was having with my Mac. I was having trouble with an SSD where fsck looped around for many times, and said incorrect number of thread records, but eventually said that everything had been repaired. However, I still couldn’t boot, I got the gray screen and the loading bar, and a power off.

Running disk utility with the SSD in an external enclosure, I saw that the disk was recognized but that the computer was unable to mount the actual partition on it. Verify disk didn’t bring up any red flags, but when I tried to unmount the disk, the computer hung and then eventually give an error. This also made it impossible to format the drive in MacOSX.

Weirdly enough, connecting that SSD to my RPi, I was able to mount it, and even browse through it. I didn’t need to recover anything, but the disk appeared to be totally functional from Linux command line. However, running fsck on Linux lead to a hang, it never finished so I just exited out of it via control+C. At this point, I used GParted in the way I described above (to create a new partition table),at which point I connected the SSD to Mac OSX, was asked to initialize the disk, and then formatted it to the GUID partition table with Mac OS Journalized and dragged over everything I needed.

So who knew, a RPi CAN do what a Mac cannot. I can’t think of any other way that I would’ve been able to format the drive so that I could restore my system onto it.

Adding Django to Pancake on the RPi

So let’s start making a real website! Django is a web framework for use with Python, which is less natural and perhaps less versatile than Web2py, but ultimately has MUCH more documentation, a strong support community, and is ultimately “the highest-level framework, providing most of the bits in one place and with the most cohesion.

One note early on is that you need to have Python, if you don’t, do: sudo apt-get update and sudo apt-get install python

So to install, download the latest version, 1.5.1 in my case, move it into whatever directory you want your Django installation to be in, and cd into it. Then, in the terminal type:tar xzvf Django* hitting Tab to complete at the * and wait a while, in my case 7 minutes, until everything unpacks. The tar file contains the package including the subdirectory structure, and is called a tarball. In the xzvf the x is for extracting the files, z is for decompressing the file, v is for verbose mode printing out its progress, and f just indicates that you are specifying the file.

You could delete the tarball once it’s done unpacking with sudo rm Django-1.5.1.tar.gz making sure you don’t use Tab completion in a way that leaves out the file extension, and leads to your unpacked directory being deleted!

cd Django*
sudo python setup.py install

And wait for everything to install. Now, although later, we might want a better database than SQLite like PostgreSQL (More on this later), the SQLite that comes with all Python versions newer than 2.5 will do it for us.

Now you can start your Django projects with django-admin.py startproject django_project_name which creates a directory with:

Want an explanation of what all these files are and how to build your Django project? I’m currently learning Django from this book, and so should you!

Hosting a Website on The RPi

As detailed here, Pancake is lightweight built for RPi server software. And it’s pretty quick to get up and running.

First, we need to install php, so I went into the terminal and typed
sudo apt-get install php5-cli

Then, I navigated to Pancake’s website and downloaded the zip of the latest version (1.4.2) I saw by clicking on it, and just clicked the “save” button for it to go into the default download directory.

Then following shows how to install Pancake on an external drive that I have mounted on the RPi already. NOTE: It is not recommended to use the defaults for installation listed on Pancake’s website because it installs onto the SD Card from which the system boots. I’ve heard of people setting up wiki’s on their RPi’s, only to kill the SD card within a week. So, I will be installing in a directory on a spin drive: /media/USBHDD/web/. Now:

unzip pp3345-Pancake-* hitting tab at the * to autocomplete, and assuming you’ve got unzip installed on the RPi, and then enter in the following commands back to back:

cd ~

cd pp3345-Pancake-*

sudo mkdir /media/USBHDD/web/Pancake

sudo cp -r * /media/USBHDD/web/Pancake

cd /media/USBHDD/web/Pancake/sys

sudo chmod +x pancake.sh

cd ..

sudo cp init /etc/init.d/pancake

sudo chmod +x /etc/init.d/pancake

sudo update-rc.d pancake defaults enable

Now, pancake will start after you reboot, that is, pancake in the “default” installation location will start on reboot. But since we’ve installed pancake on an external drive and not on the internal SD card like default says, we will need to update the pancake script with:

cd /etc/init.d/

sudo nano pancake

And change the DAEMON=/media/USBHDD/web/Pancake/sys/pancake.sh line to whatever the installation path you chose was, down to the pancake.sh file.

To start it now: sudo /etc/init.d/pancake start

If you go to http://localhost/ on the RPi, you’ll see a “Welcome to Pancake Page”
Alternatively, you could type ifconfig to find out your RPi’s IP address, and got to (or whatever that IP address is) from another computer you have still within your local network.

What you see displayed in your browser is the file /media/USBHDD/web/Pancake/vhosts/default/index.html file.

So how do you get this visible to people outside your network, and for free? More on this later. And how do you put something up on here that you want to share? Also more on this later.

Transferring/unzipping Files Mac to RPi over SSH

First, there needs to be a directory somewhere on the RPi that you have the permissions to write to. I am writing to a Mac OS Journaled external drive that is connected to the RPi (click here for how to mount a drive like that on your RPi).

Since I am on my mac laptop trying to access the RPi, I open Terminal on my mac. To make a new folder on the RPi and add the right permissions to it I ssh-ed into the RPi via: ssh -l pi (if were my IP address) and if pi were my username on the RPi, typed in my RPi account’s password, cd-ed into the right directory where I wanted the folder to be, and did:

sudo mkdir transfers && sudo chmod 777 transfers

This makes a folder called “transfers” in the current directory, and gives it writable permissions.

Then, I did exit to quit the ssh session. While you’re still in the mac’s terminal, type something like this (taken from here):

Screen Shot 2013-03-27 at 12.29.10 PM

And the file will head on over, after you are asked to type in your RPi password. Alternatively, once you have a folder with the right permissions, you could mount that folder as a network drive on your mac.

If you are transferring a zip file, make sure you have unzip and zip installed via:
sudo apt-get install zip && sudo apt-get install unzip

then, the unzip filename.zip will be enough to unzip the file as long as you are in the correct directory. Then you can use the  mv dir1 dir2  to move the directory that was unzipped to a more desired location.

Moving the Pi’s system to a larger SDcard

First, you might as well check that your new card is compatible with the RPi. As this page says, install an external (USB) SD card reader/writer (on the Pi). Put the new SD card in the external reader/writer.

df -h to see where the external SD Card was mounted. This will be the /dev/something on the left, not the /media/something on the right.

It might be a good idea to unmount, that’s umount in the terminal, and disconnect any external drives that are connected to the RPi…one small typo in the next step could compromise an entire drive!

Then, after logging in as root via sudo -s do:

dd if=/dev/mmcblk0 of=/dev/sda bs=1M

That is, if the desired location of the clone can be found at /dev/sda. Don’t put numbers in the out file location even though they may appear after doing the df -h command! Meaning that even if your drive appears as sda1 or sda2, you just type sda. These numbers indicate partitions on the SD card.

(Alternatively, the outfile could also be an .img file: of=/media/USBHDD/backup.img if you want to save the image of the current system, and maybe restore it later with dd if=/media/USBHDD/backup.img of=/dev/sda bs=1M after plugging in your sd card into your RPi.)

Anyway, the output might look something like:

3790+0 records in
3790+0 records out
3974103040 bytes (4.0 GB) copied, 339.487 s, 11.7 MB/s

then shutdown with: sudo shutdown -h now
replace the SD Card with the new one, wait until it boots, and then expand the partition with sudo raspi-config bringing up a blue menu, where you can select the option “expand root partition” and then restart when prompted, and done!

Sharing Files between RPi and Mac

Following another blog on wordpress I installed something called netatalk on my RPi with the sudo apt-get netatalk after doing a quick update via sudo apt-get update . After it installed, I wanted to make sure I could access the external Hard Drive that I had mounted on the RPi so I stopped netatalk, edited netatalk’s configuration file to add the directory that I had my external HD mounted to, and restarted netatalk via:

sudo /etc/init.d/netatalk stop
sudo nano /etc/netatalk/AppleVolumes.default

adding the line: /media "Media" near the end after ~/ "Home Directory" 

and retstarting netatalk with sudo /etc/init.d/netatalk start . Then, on my mac, I hit Command+K and typed in afp://<ip address of RPi> (the RPi’s IP address can be retrieved with the ifconfig command). And I was asked for my RPi’s username and password, and which of the two areas of the RPi I wanted to access (“Media” or “Home Directory”) after I gave it I was in!

To test it out, I transfered a 519 MB file in 82 seconds, so that’s 6.33 MB a second, which is pretty great for me! I’m tempted to configure a time machine backup over the air!

Here is a nice description of alternatives to this fabulous solution.

Fixing the IP address

By default, the RPi expects a router to tell it an IP address, and it is DHCP ready.

I needed to set the IP address in the RPi because I moved to a static IP environment. Editing the /etc/network/interfaces file (commanding: nano /etc/network/interfaces), I got rid of the iface eth0 inet dhcp line, and as instructed by a blog added:

iface eth0 inet static
address <my IP here>

and then a simple sudo reboot set everything right.

Faster/Memoryless SSH – saving a Hostname

By editing the ~/.ssh/config file on your mac (or alternate linux machine?), you can save the RPi’s ip address to type less at the terminal when you SSH into the RPi, by entering this:

Host RPi
HostName 192.168.1.whatever
User pi

You will now be able to login to the RPi from your mac by doing ssh RPi instead of ssh pi@192.168.1.whatever, so if you forget the IP address later, that’s fine.

Mounting HFS+, from Mac to RPi

Following off of this tutorial, I installed the driver for my HFS+ drive (Mac OS Journaled), first by gaining root privileges with the command sudo -s and then with the command apt-get install hfsplus (btw, apparently, Raspberry Pis dont come with a root account).

After this, still with root privileges, and before plugging in the drive, I typed  tail -f /var/log/messages to see device information as suggested in this tutorial, then I pressed Ctr+C to get a new line, plugged in the hard drive through my powered usb hub, and typed tail -f /var/log/messages again, and looked for the change.

The change I saw was:

Mar 5 15:06:36 raspberrypi kernel: [350517.390085] sd 1:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/465 GiB)
Mar 5 15:06:36 raspberrypi kernel: [350517.391125] sd 1:0:0:0: [sda] Write Protect is off
Mar 5 15:06:36 raspberrypi kernel: [350517.455957] sda: sda1 sda2
Mar 5 15:06:36 raspberrypi kernel: [350517.460939] sd 1:0:0:0: [sda] Attached SCSI disk

I then created a directory to mount the drive to, still with root privileges by typing mkdir /media/USBHDD where USBHDD is the name that I chose. Then I was able to mount the drive by typing mount -o force -t hfsplus /dev/sda2 /media/USBHDD as I found here. By typing mount I was able to see that indeed /dev/sda2 was mounted to /media/USBHDD as an hfsplus drive. Just to check that I was able to write to it, I made a folder without issue while still with root privileges.

Then I unmounted it by doing umount -t hfsplus /dev/sda2 /media/USBHDD and I got back umount: /media/USBHDD: not mounted and after doing the mount command again, sure enough the drive wasn’t listed anymore (before I did this I had to cd out of the drive within and outside of the root privileges in my shell). Under the recommendation of the first site mentioned, I wanted to run fsck check on the disk, to do this I needed to install something else as suggested here with the command sudo apt-get install hfsprogs,  and once that was installed and my disk unmounted, I used the fsck.hfsplus /dev/sda2  command, which spit back that the HFS Plus Volume is journaled, and that I could use the -f option to force checking, so I did, typing fsck.hfsplus -f /dev/sda2 after which the drive was checked, found to need a minor repair, was repaired, and checked again.

Then, just so this drive would be mounted automatically if I were to restart, under the guidance of the first source, I edited my  /etc/fstab file with nano /etc/fstab and added the line /dev/sda2     /media/USBHDD     hfsplus     rw,user,noauto     0     0   (those are tabs between the different entries). As this website tells us, the fstab file contains all the information about devices that get mounted during the startup process. However, note that I added a noauto into the options column in the table, this means that the device will be in the system, but not mounted upon startup. To mount the device, once booted type mount /media/USBHDD . But, I decided to remove this, since I will always leave the drive connected, and rarely disconnect it. The user option allows any user to mount the device. The two zeros say specify if the drive will be checked on boot, and if so in which order. Finally, I saved this file, and rebooted.

At this point, I was told that the drive was mounted as read only, and sure enough when checked with the mount command, it specified it as ro. I decided to take some advice and try changing the options in the fstab file to /dev/sda2     /media/USBHDD     hfsplus    defaults,force     0     0. This worked! I succeeded in mounting my Mac OS Journaled volume!

One last thing…

Since I am planning on using this hard drive with transmission, I need to give extra permissions to transmission like it says here. I do this by first stopping transmission by sudo /etc/init.d/transmission-daemon stop and then, the two permissions commands sudo chmod g+rw /media/USBHDD/Torrentfolder and sudo chgrp -R debian-transmission /media/USBHDD/Torrentfolder and then finally start transmission sudo /etc/init.d/transmission-daemon start and this will avoid any “Permission denied” errors in transmission!

Notes about linux

man gives information about commands, stands for manual, best to reference. Press space to page down. Press q to quit out of the man page.

Some useful navigation commands: ls, mv, cp, mkdir, cd, rmdir, cd .., pwd, clear.

info also gives information on commands, gives descriptions that are less technical but also less informations about different options. At the bottom, there is a table of contents of more information, highlight an asterix next to a subject, and hit enter to enter it. Hit to go up a level back to the TOC. You can do ctr-s to search a page, type a query, and hit to go to next entry, and then ctr-g to stop searching (this works in the emacs text editor too). To quit out of info page, hit q. Best to use to learn.

how to look up commands that you don’t know the name of? man -k <start of search>, and all commands starting with, or including, your query will be brought up (this is the same as doing apropos <search query>).

To login as root, type su, and then exit to exit.

Bash shell is what you are interacting with when you type in the terminal, the intermediary between you and the OS. Bash path in .bash_profile includes what directories the system will look in when you type commands. PATH=$PATH:<directory>:<another directory> to include directories. source .bash_profile will make that configuration file active.

Aliases: Can add alias rm='rm -i' in .bashrc for example so that every time you command rm, system assumer you mean rm -i. Then do source .bashrc to make that configuration active.


bin directory has the most often used commands in it. opt directory is where optional software is installed into. sbin has executables, but for the system, not like for normal users like the bin directory. The usr directory also has its own bin folder

Absolute (starts with a slash) versus relative.