Freitag, 25. Mai 2012

Archive mythtv recordings to mkv using mythnuv2mkv.sh

Even with the biggest hard disks your recordings will use all the available space some time. Yes, mythtv does deletes old recordings - but some of them you want to keep and archive forever.

There is a nice little script called mythnuv2mkv.sh. The original website is now down, the script has not been updated from the author in the last two years (last published version was v1.63). An updated and extended version is available at http://pastebin.com/ENMjrLK (v1.92). The script runs almost out of the box with mythtv 0.25 and below are the required steps.

Download the script and store it as /usr/bin/mythtvnuv2mkv.sh. Open the script in an editor of your choice and modify around line 2277 and locate:
cat > ${OUTBASE}_tags.xml <<-EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE Tags SYSTEM "matroskatags.dtd">
<Tags>
<Tag>
<Targets>
EOF
 and add two lines before this block:

# Remove old file if there
[ -f ${OUTBASE}_tags.xml ] && rm -f ${OUTBASE}_tags.xml 
Save the script, chown owner and group to mythtv and set sticky-group bit.

As the script is quite IO intensive I have added an old 32GB USB stick to my system. The stick becomes available as /mnt/usb-tmp and will hold archived recordings.

Living in Europe I am receiving DVB-S in three formats: 576 (SD format, old resolution), 720p (HD) and 1080p (FullHD). 16:9 and 4:3 format will be automatically identified from mythtv. For each resolution I need one user job in MythTV Backend:

User Job1: Archive 1080: LC_ALL=C /usr/bin/mythnuv2mkv.sh --jobid=%JOBID% --quality=1080 --pass=two --denoise=OFF --deblock=ON --deleterec=OFF --crop=OFF --copydir=/mnt/usb-tmp --chanid=%CHANID% --starttime=%STARTTIME%

User Job2: Archive 720: LC_ALL=C /usr/bin/mythnuv2mkv.sh --jobid=%JOBID% --quality=720 --pass=two --denoise=OFF --deblock=ON --deleterec=OFF --crop=OFF --copydir=/mnt/usb-tmp --chanid=%CHANID% --starttime=%STARTTIME%

User Job3: Archive 576: LC_ALL=C /usr/bin/mythnuv2mkv.sh --jobid=%JOBID% --quality=576 --pass=two --denoise=ON --deblock=ON --deleterec=OFF --crop=OFF --copydir=/mnt/usb-tmp --chanid=%CHANID% --starttime=%STARTTIME%

Make sure /mnt/usb-tmp is writeable for mythtv user and group. The LC_ALL=C definition in front of each command is important for mkvmerge utility which will crash without.

Now to archive a recording you can add a cutlist to a recording and use the menu bar to start one of the three defined archive jobs. Archiving takes time - on my Quad-CPU 90 minutes 720p to mkv need three hours.

Results: Usually the file size are decreased from 10 GB to 2.8 GB.

You can also tweak some defaults in the beginning of the script (ogg vs. faac audio system, mkv or other containers, number of threads etc.).

To avoid stutter while playback you may consider to add a nice -19 to all three jobs and limit the number of concurrent jobs in the backend to two.

Mittwoch, 22. Februar 2012

mythtv and NAS storage for recordings

Since 0.22 mythtv introduced the Storage Group feature: multiple single drives are used to store the recordings or archived movies. The system uses some clever routines to utilize the least utilized (in terms of IOPS or free space) for new recordings and other IO activity.

A loss of a single drive will only impact the recordings on this drive. It's not like a RAID array where a loss of a drive may impact RAID performance or redundancy.

This system is nice and clever for recordings, which can be re-recorded when a show is again in the schedule. More drives allow smooth recordings of multiple recordings at once.

But for archived recordings I prefer an external RAID array - Synology, QNAP - whatever in RAID6 mode. This solution allows to loose two drives out of the array and still have your data available - which does not replace a backup, but protects against the loss of a single drive. Using mythnuv2mkv.sh script I can archive recordings (with an impact on my power bill, as the sytem takes ages to archive HDTV recordings) on the NAS together with my pictures.

But you do not want to wake up the NAS system for every recording on a mixed frontend/backend system. The NAS spindles down when not in use - so I need a solution which actives the NAS system when I start mythfrontend via mythwelcome, but keeps it idle when just recording a show and goes back to sleep.

To do this I modified /usr/share/mythtv/mythfrontend.sh (mythbuntu 11.04) system. As this file is being overwritten with the daily nightly updates, I need a solution to keep my changes to the file.

So each change is being backuped in the same directory and named as mythfrontend.sh.YYYYMMDD (Year, Month, Day without spaces).

The following two lines in /etc/rc.local will replace with each reboot the standard script with the latest backup:

# Replace mythfrontend.sh with our version
# Backup files have .YYYYMMDD as suffix - use latest
cp -f `ls /usr/share/mythtv/mythfrontend.sh.2* | sort -r | head -1` \
      /usr/share/mythtv/mythfrontend.sh
Now I just need to have one backup file with my changes to mount the NAS filer when mythfrontend will be started via lirc and mythwelcome.

The actual start of mythfrontend.real is being initiated in the last if-block in mythfrontend.sh. Use vim +95 mythfrontend.sh to change the code to:
    if [ "$IGNORE_NOT" = "0" ]; then
        /usr/share/mythtv/mythmount.sh
        /usr/bin/mythfrontend.real "$@"
        /usr/share/mythtv/mythunmount.sh
    fi
and backup the modified mythfrontend.sh to mythfrontend.sh.YYYMMDD (replace of course YYYY, MM and DD with the actual date).

mythmount.sh and mythunmount.sh are two new scripts which mount and unmount the NAS shares to /mnt. /mnt has four prepared mount points, FSK00, FSK06, FSK12 and FSK16 (which are the age classifications in Germany, without, six years, twelve years, sixteen years).

Now create /usr/share/mythtv/mythmount.sh:
#!/bin/sh
# Mount NFS shares
for SHARE in FSK00 FSK06 FSK12 FSK16 music pictures FSK00 FSK06; do
    # Check if already mounted (multiple starts of mythfrontend)
    MOUNTED=`mount | grep "/mnt/${SHARE}" | uniq`
    # Mount if not already mounted
    [ "$MOUNTED" ] || mount /mnt/${SHARE}
done
make it executable for everyone and change ownership to mythtv:mythtv. You see that the first two shares are again in the for-list at the end - sometimes the NAS requires too long to spindle up and become available that first two mount actions will time out. Adjust as needed.

Create mythunmount.sh as well:
#!/bin/sh
# umount NFS shares
for SHARE in FSK00 FSK06 FSK12 FSK16 music pictures; do
    # Check if already mounted (multiple starts of mythfrontend)
    MOUNTED=`mount | grep "/mnt/${SHARE}" | uniq`
    # umount if already mounted
    [ "$MOUNTED" ] && umount /mnt/${SHARE}
done
 and change again ownership and executable-bits.

To automatically mount the shares you also need to prepare /etc/fstab and add a line for each share:
//192.168.1.17/movies-fsk0    /mnt/FSK00    cifs    username=mythtv,password=XXX,uid=107,gid=112,iocharset=utf8,user,noauto    0    0
//192.168.1.17/movies-fsk6    /mnt/FSK06    cifs    username=mythtv,password=XXX,uid=107,gid=112,iocharset=utf8,user,noauto    0    0
//192.168.1.17/movies-fsk12    /mnt/FSK12    cifs    username=mythtv,password=XXX,uid=107,gid=112,iocharset=utf8,user,noauto    0    0
//192.168.1.17/movies-fsk16    /mnt/FSK16    cifs    username=mythtv,password=XXX,uid=107,gid=112,iocharset=utf8,user,noauto    0    0
Of course this requires an user account named mythtv with password XXX on the NAS solution. Adjust uid and gid settings to your system and change the IP address to the one from your NAS device.

Now the shares are mounted and unmounted, but mythtv does not see any archived recordings from the NAS, as mythtv searches for files in /var/lib/mythtv/videos, not in /mnt. Just add a symlink for each share into that folder:
lrwxrwxrwx  1 root   mythtv   10 2011-03-31 22:36 FSK00 -> /mnt/FSK00
lrwxrwxrwx  1 root   mythtv   10 2011-03-31 22:36 FSK06 -> /mnt/FSK06
lrwxrwxrwx  1 root   mythtv   10 2011-03-31 22:36 FSK12 -> /mnt/FSK12
lrwxrwxrwx  1 root   mythtv   10 2011-03-31 22:36 FSK16 -> /mnt/FSK16
lrwxrwxrwx  1 root   mythtv   10 2011-03-31 22:36 FSK18 -> /mnt/FSK18
Now after a reboot you should have a working combination of NAS external storage, accessing organized archived recordings while the NAS stays quit when the mixed frontend/backend is recording a new show. Whenever a human being starts mythfrontend via mythwelcome, NAS is being mounted and archived recordings become available.

Donnerstag, 26. Januar 2012

TBS 6981 after upgrade to Kernel 3.0.0

Well,

as a linux users with a good background in security I do update and upgrade my mythbuntu box on a regular basis. Even if it's an private LAN - there are more vulnerable clients in the same LAN and they could be used as a hop to attack the mythtv box.

So, after a nice
apt-get dist-upgrade
to migrate to mythbuntu 11.10 I suddenly had Kernel 3.0.0 in use. Bad karma for my two lovely, rock solid Technotrend S3200 DVB-S2 cards. Both cards could not lock on a transponder anymore, LiveTV was gone. Within the status field on mythfrontend the transponder status became
Tl__
for about 15 seconds and then mythtv complained about a missing lock.


According to http://www.linuxtv.org/wiki/index.php/TechnoTrend_TT-budget_S2-3200#Card_only_rarely_locks_onto_any_transponder this is a known bug. Comments about the TT S3200 card were bad and I was not in the mood to re-compile a stock mythbuntu kernel.

So I searched for alternatives and found the http://linuxtv.org/wiki/index.php/TBS_6980/6981 TBS 6981 card, judging by comments the best DVB-S2 card available. And it has a PCIe interface and twin tuners - awesome!

Purchased on Amazon, delivered next day. Installation quite straightforward:
  1. Copy linux drivers from http://www.tbsdtv.com/english/Download.html
  2. Transfer to mythtv box into /tmp
  3. mkdir TBS6981 && cd TBS6981
  4. unzip ../linux-tbs-drivers-111118.zip
  5. tar -xvjf linux-tbs-drivers.tar.bz2
  6. cd linux-tbs-drivers
  7. ./v4l/tbs-x86_r3.sh (for I32 Kernel 3.0.0)
  8. make
  9. make install
  10. reboot
Great - 5 minutes of work. mythtv worked out of the box without any changes on the backend.

Best thing: the new TBS 6981 card is way more sensitive than the Technotrend S2-3200! Signal quality increased from 2% to 76%, SNR ratio from 2.0 dB to 4.8dB. What a good buy!

If you wonder about the small numbers with the TT S2 3200 card - I am using a tiny Selfsat dish.

Summary: Perfect card. Ultimate recommendation.