UPDATE: 23rd November 2024. This has now been enhanced with the handling mostly within a single python script, improved processing time and creation of a subtitles alternative output.

Even though most of our TV time is on streaming platforms these days, sometimes I watch TV programs with my wife, or F1 highlights on my own.

Often, we'll just watch the program earlier the following day, especially if it's on in the evening (my wife has to get up early).

We tend to watch Channel 4 programs the most, and sometimes ITV and the BBC. In the UK, the BBC runs programs without adverts, funded by the TV License - a unique set up in the world.

However, Channel 4 and ITV are funded by adverts and often 25% of watch time is adverts. Both have streaming solutions, but these also show adverts (unless you pay a fee) and some of them are very repetitive.

My solution is to just record the TV stream, then process it afterwards so it can be streamed locally using my home server to the TV.

The recording records the adverts, so the channels are still effectively paid for the viewing, but after processing the adverts are cut using a piece of software named comskip.

Comskip needs a configuration file to optimise its performance. My comskip.ini is here and this seems to work well on ITV and Channel 4.

This setup has taken some time of tweaks and optimisations to set up, but the result uses all free software and works quite well.

Now, my setup is a little unique. I use my server for various tasks and hosting Emby Media Server is one task so that I can stream Music and Movies to my TV, phone, PCs or tablet.

The server is at the entry to the flat, which is in the middle of the building so a bad place for TV signal and nowhere near an aerial socket. This is why I use an old PC for doing the recording. That PC runs LibreElec on bare metal well and has good built in compatibility with the USB TV Tuner Stick Geniatech MyGica T230A. Should the signal be bad, it's convenient to restart without my server going down.

The old PC is connected to an even older 26" 720p TV via VGA D-SUB and 3.5mm audio - but it's a reasonable TV and audio is handled by a homemade amp, of course 😉

In the Living Room, I have a big TV and sound system, but no PC is connected to it since I use a Roku Streaming Stick+ for the steaming services. This conveniently supports H.264 (AVC) and H.265 (HEVC) decoding at a low power. An Emby app for it connects to my server so all material is available there.

LibreElec has TVHeadend added for TV support and recording, and the Kodi addon for showing the TV Guide and managing recordings. TVHeadend does have a web interface too, but it sadly ain't pretty!

I record in native format. This allows, if I want to, to watch a TV recording from the beginning, whilst it is still recording. I often do this with F1 so I can start watching about 45 minutes later than the start time and manually skip adverts, so it finishes close to the original end time, within Kodi itself.

For streaming to the Roku or elsewhere though, I do some processing. Now, technically HD TV signal in the UK is already H.264 but signal drops mean it does not stream that well, and running comskip on the original files also did not output correct timings. The recording is also interlaced so needs de-interlacing before streaming (bwdif is used to double the framerate). So, I re-encode the recording again using FFMPEG.

This is done on the server. The complete steps are:

1. List files on the remote (recording) PC
2. For each file, check if it has been processed already. If not:
   1. Copy the file from the recording PC to the server (I also delete them from the source)
   2. Use FFMPEG to put the .ts file into a .mkv (matroska) file. This helps comskip processes the recording better.
   3. Use comskip to output the start and end times for commercials in SCF format
   4. Convert this file to a concat.txt file that FFMPEG concat demuxer can read
   5. Use FFMPEG to convert the video:
      1. FFMPEG concat demuxer to concatenate start and end times of the same source encoded file to make a shorter version
      2. Use libx264 codec to convert to H.264 / AVC format
      3. Use bwdif to deinterlace
      4. Create a second copy with burned in subtitles, for streaming
3. Shut down the recording PC

I wrote a python script to do all of the above. This is scheduled at 1AM via a cron job (crontab) running on the server.

The server has a Core i7 10700 processor which is 8 Cores / 16 Threads CPU. It does support VAPI encoding too, but the quality/speed I found not competitive compared with CPU encoding. I found the medium preset and CRF of 25 a fair speed/compression/quality tradeoff for the output files, though I now use CRF 23. The bitrate cannot be too high, otherwise the Roku can't stream it. I also wanted the processing time to be fast to minimise energy usage and make a file available sooner in case I wanted to watch the show on the same day.

I've picked H.264 as the video codec here for maximum compatibility. There is not much difference on quality and file size for 1080p TV recordings compared to the newer H.265 (which is much better at UHD), but it means my older hardware can decode it without the Emby server transcoding.

To pick a preset, I tested encoding the first 2 minutes of a recording, with a CRF set to 25, with yadif filter (I later switched to bwdif). Here are the results:

Your mileage will of course vary. Yes, slow and fast resulted in the same size, and so did superfast and ultrafast. Veryfast produced the smallest file! The picture quality between them was not much.

For de-interlacing, which is necessary for most UK DVB broadcasts, the filter makes a difference:

Both yadif=1 (yadif 2x) and bwdif double the framerate so you get a 50fps file instead of 25fps, but both strangely result in a smaller file though add to processing time. I had a slight preference for bwdif for Formula 1 car movement.

The full native .ts recording is 90 minutes and 3.6GB in size.

Re-encoding with the veryfast preset takes 13 minutes 26 seconds and produces a file size of 2.4GB.

ffmpeg -y -i "recording.ts" -vf 'bwdif' -vcodec libx264 -preset veryfast -tune film -crf 25 -acodec copy -scodec copy -f matroska "recording.mkv"

Running comskip takes 2 minutes 2 seconds.

comskip --scf --hwassist --threads=8 --ts --ini=comskip.ini "recording.mkv"

Splitting it and concatenating the file without adverts takes just 3 seconds (NVMe SSD helps here!) and produces a file size of 1.9GB, and a runtime of 67 minutes.

ffmpeg -y -f concat -safe 0 -i "recording_concat.txt" -vcodec copy -acodec copy -scodec copy "recording-cut.mkv"

Ideally, the comskip chapters file would be supported natively in mkv in the Roku Emby player - but I couldn't find a way and I instead trust comskip to do a good job and slice the file and combine it again using FFMPEG. The final file is served without adverts, however my script does not delete the originals just in case.

Unfortunately, comskip does not produce a file format that works with FFMPEG so I originally made a python script to convert it.

The full python script (updated 23rd November 2024) is below.

import sys
import os.path
import subprocess
import glob
from datetime import datetime

localdir = "/data/dan/Videos/TV"
ramdir = "/run/shm"
tempdir = "/data/dan/Videos/TV2"
remotedir = "/media/sdb1-ata-ST500LM000-1EJ16/recordings"
remoteuser = "root"
remotehost = "tv-pc.portal.home"
doshutdownafter = True
crf = 23

# Use ssh to find all the remote file recordings
remotecmd = subprocess.Popen(f'ssh {remoteuser}@{remotehost} find {remotedir} -name "*.ts"', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
remoteoutput, remoteerr = remotecmd.communicate()

if remotecmd.returncode == 0:
    
    filestoprocess = []
    # Split the found files into a list
    remotefiles = remoteoutput.splitlines()
    # Find all locally processed recordings
    localfiles = glob.glob(f'{localdir}/**/*.mkv',recursive=True)
    
    # Files we've already processed
    for localfile in localfiles:
        localfile = os.path.basename(localfile)
        
    # Files on the remote PC
    for remotefile in remotefiles:
        docopy = True
        for localfile in localfiles:
            if os.path.splitext(os.path.basename(localfile))[0] == os.path.splitext(os.path.basename(remotefile))[0]:
                # Found file locally - do not process it
                docopy = False

        if docopy:
            print(f"Will process: {remotefile}")
            filestoprocess.append(remotefile)

    # Process files
    for filetoprocess in filestoprocess:
        # Copy the remote file to RAM first
        print(f"COPYING: {filetoprocess}")
        rsynccmd = subprocess.Popen(f'rsync --remove-source-files -avh {remoteuser}@{remotehost}:"{filetoprocess}" "{ramdir}"', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
        rsyncoutput, rsyncerr = rsynccmd.communicate()
        if rsynccmd.returncode != 0:
            print(f"Could rsync file {filetoprocess}")
            print(rsyncerr)
            continue
        
        filename = os.path.basename(filetoprocess)
        filename_noext = os.path.splitext(filename)[0]

        print(f"PROCESSING: {filename}")

        # FFMPEG used to put ts into mkv, which helps comskip process accurately. This will do a quick lossless copy.
        print(f"STARTING FFMPEG...")
        #processcmd = subprocess.Popen(f'< /dev/null ffmpeg -n -i "{ramdir}/{filename}" -vf \'bwdif\' -vcodec libx264 -preset veryfast -tune film -crf 25 -acodec copy -scodec copy -f matroska "{tempdir}/{filename_noext}.mkv"', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
        processcmd = subprocess.Popen(f'< /dev/null ffmpeg -y -i "{ramdir}/{filename}" -vcodec copy -acodec copy -scodec copy -f matroska "{tempdir}/{filename_noext}.mkv"', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
        processcmdoutput, processcmderr = processcmd.communicate()
        if processcmd.returncode != 0:
            print(f"Could not convert file {filename}")
            print(processcmderr)
            continue

        print(processcmdoutput)

        # Delete ts file from RAM - no longer needed
        os.remove(f"{ramdir}/{filename}") 

        # COMSKIP
        print(f"STARTING COMSKIP...")
        processcmd = subprocess.Popen(f'/usr/local/bin/comskip --scf --hwassist --threads=8 --ts --ini={sys.path[0]}/comskip.ini "{tempdir}/{filename_noext}.mkv"', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
        processcmdoutput, processcmderr = processcmd.communicate()
        if processcmd.returncode != 0:
            print(f"Could not detect commercials on file {filename}")
            print(processcmderr)
            continue

        print(processcmdoutput)

        # Process SCF
        print(f"PROCESSING SCF...")
        filenamesafe = filename_noext.replace("'", "\'\\\'\'") + ".mkv"

        # Using readlines()
        file_scf = open(f"{tempdir}/{filename_noext}.scf", 'r')
        scf_lines = file_scf.readlines()

        file_concat = open(f"{tempdir}/{filename_noext}_concat.txt", 'w')

        # Strips the newline character
        line_str = ""
        time_str = ""
        chapter_count = 1
        inpoint = 0
        outpoint = 0

        for line in scf_lines:
            line = line.strip()
            if line.split("=")[0] == f"CHAPTER{chapter_count:02}":
                print(f"CHAPTER{chapter_count:02}")
                time_str = line.split("=")[1]
            if line.split("=")[0] == f"CHAPTER{chapter_count:02}NAME":
                if line.split("=")[1] == "Commercial starts":
                    pt = datetime.strptime(time_str, "%H:%M:%S.%f")
                    outpoint = pt.hour*3600 + pt.minute*60 + pt.second + pt.microsecond / 1000000
                    if (outpoint > 0):
                        file_concat.write(f"file '{tempdir}/{filenamesafe}'\n")
                        file_concat.write(f"inpoint {inpoint}\n")
                        file_concat.write(f"outpoint {outpoint}\n")
                if line.split("=")[1] == "Commercial ends":
                    pt = datetime.strptime(time_str, "%H:%M:%S.%f")
                    inpoint = pt.hour*3600 + pt.minute*60 + pt.second + pt.microsecond / 1000000
                chapter_count += 1

        file_concat.write(f"file '{tempdir}/{filenamesafe}'\n")
        file_concat.write(f"inpoint {inpoint}\n")

        # writing to file
        file_concat.close()
        
        # Concat and encode with deinterlacing
        print(f"STARTING FFMPEG CONCAT AND DEINTERLACE ENCODE...")
        #processcmd = subprocess.Popen(f'< /dev/null ffmpeg -y -i "{tempdir}/{filename_noext}-cut.mkv" -vf \'bwdif\' -vcodec libx264 -preset veryfast -tune film -crf {crf} -acodec copy -scodec copy "{localdir}/{filename_noext}.mkv"', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
        processcmd = subprocess.Popen(f'< /dev/null ffmpeg -y -f concat -safe 0 -i "{tempdir}/{filename_noext}_concat.txt" -vf \'bwdif\' -vcodec libx264 -preset veryfast -tune film -crf {crf} -acodec copy -scodec copy "{localdir}/{filename_noext}.mkv"', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
        processcmdoutput, processcmderr = processcmd.communicate()
        if processcmd.returncode != 0:
            print(f"Could convert file {filename}")
            print(processcmderr)
            continue

        print(processcmdoutput)

        
        # Concat and create a burned in subtitle version, with deinterlacing
        print(f"STARTING FFMPEG CONCAT, DEINTERLACE AND SUBTITLE BURN IN...")
        #processcmd = subprocess.Popen(f'< /dev/null ffmpeg -y -i "{tempdir}/{filename_noext}-cut.mkv" -vcodec libx264 -preset veryfast -tune film -crf {crf} -filter_complex "[0:v]bwdif[0v];[0v][0:s]overlay[v]" -map "[v]" -map 0:a "{localdir}/{filename_noext}-subtitles.mkv"', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
        processcmd = subprocess.Popen(f'< /dev/null ffmpeg -y -f concat -safe 0 -i "{tempdir}/{filename_noext}_concat.txt" -vcodec libx264 -preset veryfast -tune film -crf {crf} -filter_complex "[0:v]bwdif[0v];[0v][0:s]overlay[v]" -map "[v]" -map 0:a "{localdir}/{filename_noext}-subtitles.mkv"', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
        processcmdoutput, processcmderr = processcmd.communicate()
        if processcmd.returncode != 0:
            print(f"Could convert file {filename}")
            print(processcmderr)
            continue

        print(processcmdoutput)


        print(f"COMPLETED {filetoprocess}")

    # Shutdown remote server
    if doshutdownafter:
        print("Shutting down remote PC")
        remoteoffcmd = subprocess.Popen(f'ssh {remoteuser}@{remotehost} poweroff', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
        remoteoffoutput, remoteofferr = remoteoffcmd.communicate()

else:
    print("Could not open remote system")
    print(remoteerr)

During running, this converts the following example scf:

CHAPTER01=00:00:00.001
CHAPTER01NAME=Commercial starts
CHAPTER02=00:00:06.000
CHAPTER02NAME=Commercial ends
CHAPTER03=00:13:20.017
CHAPTER03NAME=Commercial starts
CHAPTER04=00:17:28.001
CHAPTER04NAME=Commercial ends
CHAPTER05=00:25:28.022
CHAPTER05NAME=Commercial starts
CHAPTER06=00:29:35.021
CHAPTER06NAME=Commercial ends
CHAPTER07=00:39:00.000
CHAPTER07NAME=Commercial starts
CHAPTER08=00:43:11.024
CHAPTER08NAME=Commercial ends
CHAPTER09=00:52:19.020
CHAPTER09NAME=Commercial starts
CHAPTER10=00:56:32.016
CHAPTER10NAME=Commercial ends
CHAPTER11=01:15:12.006
CHAPTER11NAME=Commercial starts
CHAPTER12=01:19:21.024
CHAPTER12NAME=Commercial ends
CHAPTER13=01:28:27.003
CHAPTER13NAME=Commercial starts
CHAPTER14=01:30:09.001
CHAPTER14NAME=Commercial ends

To the following recording_concat.txt:

file 'recording.mkv'
inpoint 6
outpoint 800.017
file 'recording.mkv'
inpoint 1048.001
outpoint 1528.022
file 'recording.mkv'
inpoint 1775.021
outpoint 2340
file 'recording.mkv'
inpoint 2591.024
outpoint 3139.020
file 'recording.mkv'
inpoint 3392.016
outpoint 4512.006
file 'recording.mkv'
inpoint 4761.024
outpoint 5307.003
file 'recording.mkv'
inpoint 5409.001

This txt format with the same file name but different in and out points can be used by ffmpeg to combine those sections into one file.

The bash script to call it is now simple:

#!/bin/sh
echo $(date) >> /data/dan/Videos/Scripts/TVProcess.log
/usr/bin/python3 /data/dan/Videos/Scripts/TVProcess.py >> /data/dan/Videos/Scripts/TVProcess.log 2>&1

This script is scheduled daily using crontab -e:

0 1 * * * /data/dan/Videos/Scripts/TVProcess.sh

The directories are all configured in the python script, so update those as required.

You'll not be able to use the script as-is. If you are copying from another machine, you'll need to set up password-less SSH (ssh-keygen followed by ssh-copy-id) and correct the source directory as well as destination directory customisations.

The comskip.ini file must be in the same location.

What's not working?

Subtitles are limited. They are retained in DVB format in the conversion to mkv, but DVB subtitles are not supported by Roku Emby media player and converting DVB to text compatible SRT (sub rip) format is hard to automate. Therefore, a second encode is done via ffmpeg which burns in the subtitles to the main video. When we play a recording, we pick the one with the file name ending with -subtitles if we want to watch with them.

5.1 sound (AAC) is not working either. Everything I've recorded doesn't have it available in the original stream and it might be a limitation of the USB TV stick, PC or TVheadend.

• Emby Media Server
• LibreElec
• github.com - comskip
• superuser.com - Why does preset "veryfast" in FFmpeg generate the most compressed file compared to all other presets?
• superuser.com - How can I remove multiple segments from a video using FFmpeg?
• kaashoek.com comskip - UK Freeview