Raspberry Pi DVR - Addendum

I recently completed my Raspberry Pi DVR and put it into regular use.  But I quickly realized something was wrong.  Some recordings would work, but most recordings would fail 10, 15, 20 minutes in.  Every time it failed I noticed the same thing, if I SSH into the Pi and looked at the running processes, tvheadend was no longer running.  The mere fact that I can SSH in to the Pi means that the Pi itself and Linux were still running.  So what is happening, is tvheadend crashing?

The Problem
I started with the assumption that tvheadend was crashing, after all that seems the most logical explanation.  But where do I go from there?  I researched Linux logging and diagnostics to see if the system recorded more info about this problem.  Eventually I stumbled across /var/log/kern.log which is the log file for the Linux kernel.  Every time there was a failed recording, the following was logged:

Oct 20 20:10:00 garagepi kernel: [79819.240765] Out of memory: Kill process 2314 (tvheadend) score 922 or sacrifice child
Oct 20 20:10:00 garagepi kernel: [79819.240785] Killed process 2314 (tvheadend) total-vm:607540kB, anon-rss:404384kB, file-rss:384kB

Aha!  So the problem here is not that tvheadend is crashing, the problem is Linux ran out of memory so to keep the system up and running, it killed the process using the most RAM which was tvheadend.

Because the Raspberry Pi only has a paltry 512MB of RAM, tvheadend is able to consume that and more resulting in this problem.  But if we dig deeper, what is going on here?  Does tvheadend have poor memory management?  Does tvheadend have memory leaks?  Is the task of recording live TV just too much for the limited resources of Raspberry Pi?  Or is there something else going on here?  I think the answer is mostly in the "something else going on here" category.

Let us start off by looking at what tvheadend is doing.  Basically all it is doing is shuttling data from one pipe to another.  The "input" pipe is the digital TV stream coming in from the TV tuner.  For ATSC TV, this amounts to approximately 5GB of data per hour.  That data is then written (unmodified) to the "output" file.  In my setup, the output file is located on my NAS box, so the data must be transmitted over wifi to get there.  And therein lies the problem.  What if the Raspberry Pi cannot transmit that 5GB/hour of data over wifi fast enough?  Linux, like all modern operating systems, would allocate more buffers to hold that data until such time that it is written over wifi.  This means if wifi is too slow, tvheadend will continue to consume more and more system memory in the form of output buffers until such time as Linux steps in and shuts it down.

The Solution
Now that we know the problem, what can we do about it?  The short answer is, we need to improve the memory "situation" of the Raspberry Pi.  For this I made 3 major changes.

1. The Raspberry Pi has 512MB of RAM, which is split between both the CPU and GPU.  By default the GPU takes 64MB of RAM.  Since this is a headless unit in my garage, let us change that so the GPU gets the bare minimum.  Editing the config.txt file, set "gpu_mem_512=16"  This gives 16MB to the GPU which is the minimum.  This frees up 48MB of RAM for the CPU.  One downside is the GPU requires more memory than 16 for HDMI output.  I have read that either 24 or 32 is the minimum required for HDMI output.  So if you want HDMI output then set this value accordingly.  Either way, this setting is the single largest way to free up memory.  No other step will come close to the 48MB this frees up.
2. Second step is to stop any unnecessary background processes.  I am no Linux expert, so it was hard for me to know what to stop and how to stop it.  But doing some research, everyone suggests turning off the getty processes.  To do this edit the file /etc/inittab.  Scroll down and look for the "getty" lines.  Disable the getty lines by placing a '#' in front of that line.  Reboot and they are not disabled.
3. The final step is to do anything possible to improve wifi speeds.  Faster wifi means fewer buffers.  On Raspbian there are at least 2 ways to check your wifi connection details.  The first is "iwconfig wlan0" and the second is "cat /proc/net/wireless"  Both show two numbers, the link quality and signal level - both measured 0 to 100.  Both numbers are important.  What you want to do is adjust the position of your wifi antenna to increase these numbers.  My starting signal level was low 50s, after adjusting the antenna I was able to reach the high 80s.  Just that little bit made a huge difference!  Timing the copy of a large file across wifi, with signal in the 50s it took 1 minute and 29 seconds.  With signal in the 80s that dropped to 4 seconds!  So without a doubt, wifi antenna placement is critical.  Before you start fine tuning your wifi, run the command "watch -n 1 cat /proc/net/wireless"  This will display the current wifi settings updated once a second.  This makes it easy to see signal levels as you move the antenna.

Conclusions
If you wish to undertake a project like this and turn your Raspberry Pi into a DVR, there are several things that I consider crucial to your success.

First, make the above changes to give the minimum memory to the GPU and shutdown any unnecessary processes.

Second, if at all possible, use the ethernet jack instead of wifi.  Wired is always going to be faster and more reliable.  So only use wifi if you have no other option.

Third, if you do use wifi be sure to turn your wifi for maximum signal strength and speed.  Consider one of those homemade wifi antenna reflectors (plans available on the Internet) to improve wifi signal.

Fourth, with wifi optimized if you still run into problems, consider recording the video locally.  You can record the video directly to the SD card or onto a USB thumb drive or external hard drive plugged into the USB port.  Whereas it is not ideal to record large files directly to flash memory, you may have no choice.  After recording you can always copy the file over wifi during idle time when it is no longer a time sensitive process.

Fifth, consider frequent reboots.  Linux is supposed to be stable over the long run, but this does not mean all daemons (like tvheadend) are.  I think I am going to set up a cron job to reboot the Raspberry Pi every night.  After all, it is not doing anything else at 3 in the morning, so why not reboot and help tvheadend to free up memory?