Freitag, 14. Juni 2013

Validating audio and subtitle track availability in collections

Archiving a full set of episodes of a TV-series has some challenges: missing episodes, bad quality, recording issues - and language and subtitles. You may have missed one episode on your preferred HDTV channel and got one in SD lacking subtitles and original language.

Below script (yes, I know it's ugly) will dump language information for audio tracks and subtitle tracks from a MKV file in a sorted order. This allows a quick and easy check if your archived episodes have all the same language information.

#!/usr/bin/perl

while (<>) {
last unless /\S/;
$lines .= $_;
}

@tracks = split /A track/, $lines;

@audiotracks = grep/type: audio/, @tracks;
@subtracks = grep /type: subtitles/, @tracks;

foreach (@audiotracks) {
push(@atracks, $1) if / Language: (\w{3})/;
}
foreach (@subtracks) {
push(@stracks, $1) if / Language: (\w{3})/;
}

print "Audiotracks:\t";
foreach (sort(@atracks)) {
print "$_ ";
}
print "\n";

print "Subtitles:\t";
foreach (sort(@stracks)) {
print "$_ ";
}
print "\n";
Save this file as /usr/bin/mkvinfo.pl and make it executable. Move to the folder holding the archived MKV recordings and use a simple wrapper script:

for i in *.mkv; do
        echo "Checking $i"
        mkvinfo "$i" | /usr/bin/mkvinfo.pl
done
 
As result you will get a list of filenames (your mkv files) and indented a list of audio and subtitles tracks. By visual comparison you can easily spot a missing track and re-record that episode with a Record task in MythTV.

Freitag, 25. Mai 2012

Archive mythtv recordings to mkv using mythnuv2mkv.sh

Even with the biggest hard disks your recordings will use all the available space some time. Yes, mythtv does deletes old recordings - but some of them you want to keep and archive forever.

There is a nice little script called mythnuv2mkv.sh. The original website is now down, the script has not been updated from the author in the last two years (last published version was v1.63). An updated and extended version is available at http://pastebin.com/ENMjrLK (v1.92). The script runs almost out of the box with mythtv 0.25 and below are the required steps.

Download the script and store it as /usr/bin/mythtvnuv2mkv.sh. Open the script in an editor of your choice and modify around line 2277 and locate:
cat > ${OUTBASE}_tags.xml <<-EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE Tags SYSTEM "matroskatags.dtd">
<Tags>
<Tag>
<Targets>
EOF
 and add two lines before this block:

# Remove old file if there
[ -f ${OUTBASE}_tags.xml ] && rm -f ${OUTBASE}_tags.xml 
Save the script, chown owner and group to mythtv and set sticky-group bit.

As the script is quite IO intensive I have added an old 32GB USB stick to my system. The stick becomes available as /mnt/usb-tmp and will hold archived recordings.

Living in Europe I am receiving DVB-S in three formats: 576 (SD format, old resolution), 720p (HD) and 1080p (FullHD). 16:9 and 4:3 format will be automatically identified from mythtv. For each resolution I need one user job in MythTV Backend:

User Job1: Archive 1080: LC_ALL=C /usr/bin/mythnuv2mkv.sh --jobid=%JOBID% --quality=1080 --pass=two --denoise=OFF --deblock=ON --deleterec=OFF --crop=OFF --copydir=/mnt/usb-tmp --chanid=%CHANID% --starttime=%STARTTIME%

User Job2: Archive 720: LC_ALL=C /usr/bin/mythnuv2mkv.sh --jobid=%JOBID% --quality=720 --pass=two --denoise=OFF --deblock=ON --deleterec=OFF --crop=OFF --copydir=/mnt/usb-tmp --chanid=%CHANID% --starttime=%STARTTIME%

User Job3: Archive 576: LC_ALL=C /usr/bin/mythnuv2mkv.sh --jobid=%JOBID% --quality=576 --pass=two --denoise=ON --deblock=ON --deleterec=OFF --crop=OFF --copydir=/mnt/usb-tmp --chanid=%CHANID% --starttime=%STARTTIME%

Make sure /mnt/usb-tmp is writeable for mythtv user and group. The LC_ALL=C definition in front of each command is important for mkvmerge utility which will crash without.

Now to archive a recording you can add a cutlist to a recording and use the menu bar to start one of the three defined archive jobs. Archiving takes time - on my Quad-CPU 90 minutes 720p to mkv need three hours.

Results: Usually the file size are decreased from 10 GB to 2.8 GB.

You can also tweak some defaults in the beginning of the script (ogg vs. faac audio system, mkv or other containers, number of threads etc.).

To avoid stutter while playback you may consider to add a nice -19 to all three jobs and limit the number of concurrent jobs in the backend to two.

Mittwoch, 22. Februar 2012

mythtv and NAS storage for recordings

Since 0.22 mythtv introduced the Storage Group feature: multiple single drives are used to store the recordings or archived movies. The system uses some clever routines to utilize the least utilized (in terms of IOPS or free space) for new recordings and other IO activity.

A loss of a single drive will only impact the recordings on this drive. It's not like a RAID array where a loss of a drive may impact RAID performance or redundancy.

This system is nice and clever for recordings, which can be re-recorded when a show is again in the schedule. More drives allow smooth recordings of multiple recordings at once.

But for archived recordings I prefer an external RAID array - Synology, QNAP - whatever in RAID6 mode. This solution allows to loose two drives out of the array and still have your data available - which does not replace a backup, but protects against the loss of a single drive. Using mythnuv2mkv.sh script I can archive recordings (with an impact on my power bill, as the sytem takes ages to archive HDTV recordings) on the NAS together with my pictures.

But you do not want to wake up the NAS system for every recording on a mixed frontend/backend system. The NAS spindles down when not in use - so I need a solution which actives the NAS system when I start mythfrontend via mythwelcome, but keeps it idle when just recording a show and goes back to sleep.

To do this I modified /usr/share/mythtv/mythfrontend.sh (mythbuntu 11.04) system. As this file is being overwritten with the daily nightly updates, I need a solution to keep my changes to the file.

So each change is being backuped in the same directory and named as mythfrontend.sh.YYYYMMDD (Year, Month, Day without spaces).

The following two lines in /etc/rc.local will replace with each reboot the standard script with the latest backup:

# Replace mythfrontend.sh with our version
# Backup files have .YYYYMMDD as suffix - use latest
cp -f `ls /usr/share/mythtv/mythfrontend.sh.2* | sort -r | head -1` \
      /usr/share/mythtv/mythfrontend.sh
Now I just need to have one backup file with my changes to mount the NAS filer when mythfrontend will be started via lirc and mythwelcome.

The actual start of mythfrontend.real is being initiated in the last if-block in mythfrontend.sh. Use vim +95 mythfrontend.sh to change the code to:
    if [ "$IGNORE_NOT" = "0" ]; then
        /usr/share/mythtv/mythmount.sh
        /usr/bin/mythfrontend.real "$@"
        /usr/share/mythtv/mythunmount.sh
    fi
and backup the modified mythfrontend.sh to mythfrontend.sh.YYYMMDD (replace of course YYYY, MM and DD with the actual date).

mythmount.sh and mythunmount.sh are two new scripts which mount and unmount the NAS shares to /mnt. /mnt has four prepared mount points, FSK00, FSK06, FSK12 and FSK16 (which are the age classifications in Germany, without, six years, twelve years, sixteen years).

Now create /usr/share/mythtv/mythmount.sh:
#!/bin/sh
# Mount NFS shares
for SHARE in FSK00 FSK06 FSK12 FSK16 music pictures FSK00 FSK06; do
    # Check if already mounted (multiple starts of mythfrontend)
    MOUNTED=`mount | grep "/mnt/${SHARE}" | uniq`
    # Mount if not already mounted
    [ "$MOUNTED" ] || mount /mnt/${SHARE}
done
make it executable for everyone and change ownership to mythtv:mythtv. You see that the first two shares are again in the for-list at the end - sometimes the NAS requires too long to spindle up and become available that first two mount actions will time out. Adjust as needed.

Create mythunmount.sh as well:
#!/bin/sh
# umount NFS shares
for SHARE in FSK00 FSK06 FSK12 FSK16 music pictures; do
    # Check if already mounted (multiple starts of mythfrontend)
    MOUNTED=`mount | grep "/mnt/${SHARE}" | uniq`
    # umount if already mounted
    [ "$MOUNTED" ] && umount /mnt/${SHARE}
done
 and change again ownership and executable-bits.

To automatically mount the shares you also need to prepare /etc/fstab and add a line for each share:
//192.168.1.17/movies-fsk0    /mnt/FSK00    cifs    username=mythtv,password=XXX,uid=107,gid=112,iocharset=utf8,user,noauto    0    0
//192.168.1.17/movies-fsk6    /mnt/FSK06    cifs    username=mythtv,password=XXX,uid=107,gid=112,iocharset=utf8,user,noauto    0    0
//192.168.1.17/movies-fsk12    /mnt/FSK12    cifs    username=mythtv,password=XXX,uid=107,gid=112,iocharset=utf8,user,noauto    0    0
//192.168.1.17/movies-fsk16    /mnt/FSK16    cifs    username=mythtv,password=XXX,uid=107,gid=112,iocharset=utf8,user,noauto    0    0
Of course this requires an user account named mythtv with password XXX on the NAS solution. Adjust uid and gid settings to your system and change the IP address to the one from your NAS device.

Now the shares are mounted and unmounted, but mythtv does not see any archived recordings from the NAS, as mythtv searches for files in /var/lib/mythtv/videos, not in /mnt. Just add a symlink for each share into that folder:
lrwxrwxrwx  1 root   mythtv   10 2011-03-31 22:36 FSK00 -> /mnt/FSK00
lrwxrwxrwx  1 root   mythtv   10 2011-03-31 22:36 FSK06 -> /mnt/FSK06
lrwxrwxrwx  1 root   mythtv   10 2011-03-31 22:36 FSK12 -> /mnt/FSK12
lrwxrwxrwx  1 root   mythtv   10 2011-03-31 22:36 FSK16 -> /mnt/FSK16
lrwxrwxrwx  1 root   mythtv   10 2011-03-31 22:36 FSK18 -> /mnt/FSK18
Now after a reboot you should have a working combination of NAS external storage, accessing organized archived recordings while the NAS stays quit when the mixed frontend/backend is recording a new show. Whenever a human being starts mythfrontend via mythwelcome, NAS is being mounted and archived recordings become available.

Donnerstag, 26. Januar 2012

TBS 6981 after upgrade to Kernel 3.0.0

Well,

as a linux users with a good background in security I do update and upgrade my mythbuntu box on a regular basis. Even if it's an private LAN - there are more vulnerable clients in the same LAN and they could be used as a hop to attack the mythtv box.

So, after a nice
apt-get dist-upgrade
to migrate to mythbuntu 11.10 I suddenly had Kernel 3.0.0 in use. Bad karma for my two lovely, rock solid Technotrend S3200 DVB-S2 cards. Both cards could not lock on a transponder anymore, LiveTV was gone. Within the status field on mythfrontend the transponder status became
Tl__
for about 15 seconds and then mythtv complained about a missing lock.


According to http://www.linuxtv.org/wiki/index.php/TechnoTrend_TT-budget_S2-3200#Card_only_rarely_locks_onto_any_transponder this is a known bug. Comments about the TT S3200 card were bad and I was not in the mood to re-compile a stock mythbuntu kernel.

So I searched for alternatives and found the http://linuxtv.org/wiki/index.php/TBS_6980/6981 TBS 6981 card, judging by comments the best DVB-S2 card available. And it has a PCIe interface and twin tuners - awesome!

Purchased on Amazon, delivered next day. Installation quite straightforward:
  1. Copy linux drivers from http://www.tbsdtv.com/english/Download.html
  2. Transfer to mythtv box into /tmp
  3. mkdir TBS6981 && cd TBS6981
  4. unzip ../linux-tbs-drivers-111118.zip
  5. tar -xvjf linux-tbs-drivers.tar.bz2
  6. cd linux-tbs-drivers
  7. ./v4l/tbs-x86_r3.sh (for I32 Kernel 3.0.0)
  8. make
  9. make install
  10. reboot
Great - 5 minutes of work. mythtv worked out of the box without any changes on the backend.

Best thing: the new TBS 6981 card is way more sensitive than the Technotrend S2-3200! Signal quality increased from 2% to 76%, SNR ratio from 2.0 dB to 4.8dB. What a good buy!

If you wonder about the small numbers with the TT S2 3200 card - I am using a tiny Selfsat dish.

Summary: Perfect card. Ultimate recommendation.

Montag, 3. Oktober 2011

mplayer: switching between languages via lirc/remote

Do you love original languages but need the translation?

Well, mplayer is able to switch between audio tracks (e.g. languages) during playback together with lirc.
Opposite to mythtv and mythfrontend mplayer requires commands, not keystrokes. Also I noticed that sometimes with frequent changes mplayer likes get the hiccups - e.g. loosing audio completely or crashes.

But here is the solution. Edit the mplayer-related lirc file (which is /home/account/.lirc/mplayer) in mythbuntu and add a section:

begin
    remote = Technotrend
    prog = mplayer
    button = KEY_REPEAT
    config = switch_audio
    repeat = 0
    delay = 0
end

KEY_REPEAT is the remote key to be used to toggle between languages and defined in /etc/lircd.conf.

MythTV - My Setup

The 1st Post

It's now almost five year since I started to use MythTV - based on Debian and Mythbuntu - as my personal PVR solution. Over the last five years my setup changed frequently and I had good and quite bad experiences with MythTV. But I thought now that it make sense to share some of the tips, gotchas and solutions I found over the years...

For all of you who are not really familiar with MythTV - it's a Linux based personal TV/recording/movie playback solution. A nightmare to setup, but once it's working it's just a dream...

My current setup is a combined backend and frontend solution (as I do not have a basement to keep the backend somewhere else). Based on a Quad-Core at 2.83 GHz (M-version) with 4 GB of RAM it has one single 160 GB SSD for operating system and 2 x 2 TB for TV recording - this totals to about 4.000hrs of standard SD television or about 1.500hrs of HDTV television. The system is tied to a Denon AVR 1709 DTS Receiver and Teufel Boxen and a 50 inch plasma TV.

Some archived recordings are stored on an external QNAP NAS filer - I prefer RAID5 or RAID6 over storing data on single drives combined with the MythTV backend.