Cypheros Transportstream Forum

English-speaking Support => TS-Doctor 2.x => Thema gestartet von: jaydear am September 04, 2018, 02:51:57

Titel: Colours disappearing from subtitles
Beitrag von: jaydear am September 04, 2018, 02:51:57
I've set TSD to keep the subtitles in my edited files but the colour information is getting lost when I compress them. I use VidCoder (a Handbrake GUI) but it cannot even 'see' the subtitles in the TS files which is a major nuisance. If I play the same TS files in PotPlayer or VLC the subtitles are there in perfect timing and with colours intact.

To get around this I use VidCoder to convert the TS files to mp4 format. Then I use MKVToolNix (v26) to add the subtitles from the TS file and the video + audio from the mp4 file which are then multiplexed together (really fast!) as an mkv file. The timing is still 100%, but the colours are gone.

I don't know if this a bug in MKVToolNix or if I'm setting something incorrectly in TSD and/or MKVToolnix. I've tried changing between ANSI and UTF8, but that didn't make any difference. Any suggestions?
Titel: Re: Colours disappearing from subtitles
Beitrag von: Mam am September 04, 2018, 07:02:10
Hmm, I guess, you must have some wrong settings in TSD somewhere.

The "colours" really do not exist, they are just HTML formatting codes (plain text). So no program in your chain really interpretes them, but just adds the text to the stream.

Therefor I guess, the colors never existed. Either you have turned them off (or, looking from the default values of TSD, never have turned them on) or, simply, there are none broadcasted!
(not every station uses colors)
Just open the SRT File from TSD with notepad and see if you find any color commands.

Btw, you can save some time and work by dropping vidcoded and using handbrake directly. There you can add the subs into the conversion process, you dont need to add them later on with mkvtoolnix anymore.

Make sure you use the same settings (ANSI/UTF8) for the charsets in both programs! English makes no big difference, but when it comes to other languages, it is very important.
Titel: Re: Colours disappearing from subtitles
Beitrag von: jaydear am September 06, 2018, 04:58:49
I guess I didn't make myself clear... "If I play the same TS files in PotPlayer or VLC the subtitles are there in perfect timing and with colours intact." means that the particular TS files containing coloured subtitles and which I have edited DO display colours when played in PotPlayer and VLC. Yes, they are present in the srt files too, but are not useful to me in my workflow.

So the colour information DOES exist! As you probably know, it is used to distinguish the speech from multiple speakers and is essential for hearing impaired viewers to avoid confusion. In my location, the majority of programs have subtitles and are coloured if necessary.

Changing charsets in TSD and MKVToolnix makes zero difference to the problem so I'm thinking that there's a bug in MKVMerge that is causing it to strip out the colour commands in the TS files that have them. Has anybody here experienced this issue?

FYI, VidCoder supports all of HB's options, plus a lot more functionality. HB on it's own is too limited and too cumbersome.
Titel: Re: Colours disappearing from subtitles
Beitrag von: Mam am September 06, 2018, 07:53:33
Zitat von: jaydear am September 06, 2018, 04:58:49
Yes, they are present in the srt files too, but are not useful to me in my workflow.

So the colour information DOES exist! As you probably know, it is used to distinguish the speech from multiple speakers and is essential for hearing impaired viewers to avoid confusion.

Aaah I see now, you are using the DVB Subs, not the SRT files derived from Teletext!
I'm not sure (that means  I am pretty much sure that they NOT) if they contain color infos at all. Yeah I know, you SEE the colors, but that is only graphics.
This is later on processed by an OCR and converted into plain text. I guess the final text does not contain any colors.

Or, if it does, it is your vidcoder that ignores them whilst generating graphics again from the text.

Why dont you use the SRT Files? they do the same and can include formatting options. You dont need to process them in vidcode, just mux them in with mkvtoolkit as they are, the player later on will convert them to display graphics on the fly (usually INCLUDING colors, some players dont).
Titel: Re: Colours disappearing from subtitles
Beitrag von: jaydear am September 06, 2018, 08:59:38
To explain... For many years I used Womble DVD, then VideoRedo to edit. A few years ago I discovered TSD v1 when I needed something to fix some glitchy files. It works very well and I really like it's unique EPG features. It is my starting point for all TS editing now - BUT - it is not frame-accurate. I use it to prepare all my TS files for VRD which is frame-accurate. Womble was frame-accurate too, but is no longer updated.

The edited files from VRD still have the sub-titles intact with their colours, so I could reload them in TSD and just generate an srt file to be muxed in later, but it all adds time and steps to do all this stuff so it's just not practical. The way I'm working now is efficient and logical (to me).

In MKVToolNix, the subs in my TS files are identified as "SubRip/SRT" which I understand to be just a text stream in the TS file, not graphics. This is borne out by the fact that it is simple to assign any font at any size at any position in PotPlayer or VLC and they do it in real time, in colour if present, even on a slowish PC.

I'm not generating srt files from TSD, all I'm doing is making it "Keep All" subtitles. I'm not extracting or processing Teletext or DVB subs in TSD. The super-speed at which MKVToolNix muxes an mkv file from the SubRip/SRT in a TS and A+V from an mp4 also suggests to me that no OCR is taking place in it.
Titel: Re: Colours disappearing from subtitles
Beitrag von: Mam am September 06, 2018, 09:53:46
Zitat von: jaydear am September 06, 2018, 08:59:38
I'm not generating srt files from TSD, all I'm doing is making it "Keep All" subtitles. I'm not extracting or processing Teletext or DVB subs in TSD.

;D I see  8)

But then I am afraid, you are asking in the wrong forum. If you dont use TSD for Subs at all, nobody here will be able to guess what is going wrong at your side.

The only thing we can assure you is that "keep all" will keep everything (including color commands).

But since SRT Files are just plain text, you can check them out with any editor like notepad or so. See, if they contain html formatting codes like bold or color commands, they can easily be made out.
If not, take a closer look at the settings of the program you use to extract the Subs with (if TSD just passes, there must be another one used to decode the Subs to SRT), if yes, try a different player (although, we all know VLC CAN show formatting commands... and if a player cannot, it usually shows the commands themselfs with the subs because it takes them for plain text).

(
I tried to find out something about VideReDo and Subs, but the only thing was about VRD4 back in 2011, so I suspect this info is not recent anymore? If still, take a look at this thread (https://www.videoredo.net/msgBoard/showthread.php?27907-Teletext-and-subtitle-supportl), they say VRD can only handle DVB Subs and the only thing it does to them is to keep them (which wont work if you change resolution of the generated video)
They say "Teletext is not supported at all", so I am wondering, how you get the SRT Files created ?
)

Titel: Re: Colours disappearing from subtitles
Beitrag von: ErichV am September 07, 2018, 23:04:47
Zitat von: Mam am September 06, 2018, 09:53:46
(
I tried to find out something about VideReDo and Subs, but the only thing was about VRD4 back in 2011, so I suspect this info is not recent anymore? If still, take a look at this thread (https://www.videoredo.net/msgBoard/showthread.php?27907-Teletext-and-subtitle-supportl), they say VRD can only handle DVB Subs and the only thing it does to them is to keep them (which wont work if you change resolution of the generated video)
They say "Teletext is not supported at all", so I am wondering, how you get the SRT Files created ?
)

The creation of *.srt files is not supported by VideoReDo. The teletext stream is kept untouched by processing the *.ts file (https://www.videoredo.net/msgBoard/showthread.php?35179-Convert-a-TS-file-with-teletext-subtitles-to-a-MKV-or-MP4-with-embedded-subtitles).
Therefore, additional tools are necessary to extract/create *.srt subtitles (e.g. TS-Doctor's TS-remuxer for remuxing the final *.ts file to *.mkv including *.srt subtitles).
Titel: Re: Colours disappearing from subtitles
Beitrag von: jaydear am September 08, 2018, 00:09:58
Hi ErichV :D Thanks for helping. I've made some progress, if it can be called that  ::)

The TS-remuxer used by TSD is actually MKVMerge which is in MKVToolNix, and it is MKVToolnix that I found is very successful and quick at muxing in the subtitles from my TS files from TSD and/or VRD5, but only in glorious monochrome.

So I posted my problem to the MKVToolnix forum on GitLab and this is the rather terse reply I got back from Moritz Bunkus... "The teletext-to-SRT converter built into mkvmerge doesn't support colors. I have no intention on working on that. If you want colors to be preserved, you'll have to use a different program for converting the teletext subtitles. I wouldn't know which program as I have no experience with such conversions outside of mkvmerge."  :(

Great - NOT! There was also a reply suggesting a program called CCExtractor which I have attempted to use, but I'm not adept with CLI-driven software with complicated paths and commands to be typed in each and every time, and so far it has confounded me. I think that's the end of my quest for any sort of efficient colour subtitle preservation - at least for now and until some programming genius can make it easy.
Titel: Re: Colours disappearing from subtitles
Beitrag von: Cypheros am September 08, 2018, 00:38:21
TS-Doctor preserves color information for Teletext and DVB subtitles on converting to srt.
Titel: Re: Colours disappearing from subtitles
Beitrag von: jaydear am September 08, 2018, 01:48:14
I know 8)  Problem for me is that it's another step to rescan the finished TS file, then another step to remux the srt file and the mp4 that's standing by for subs. I know that's only one more step, but it takes time and I have many files to do.

The big hole in my process is Handbrake. It can't even see the subs in the TS file! If that worked I could reduce the steps and the time it all takes and avoid using MKVToolnix altogether. But I'm not going to bash my head against Handbrake's brick wall.

It'd also be nice if MKVToolNix would make an SRT file instead of an MKS file, but there's a monochrome brick wall there anyway ;D
Titel: Re: Colours disappearing from subtitles
Beitrag von: Mam am September 08, 2018, 17:44:42
Hmm, I still dont understand the fuzz you are doing ;D

Now we know, the colors are dropped by mkvtoolnix, they say, they wont fix it. So use a differnent program.
No need to search very far, TSD does the job nicely and without any problems.

I cannot understand why you are refusing to use it ?!?!?

And why are you now talking about Handbrake? before you said, you dont like it and use VideoReDo instead (which is more or less just a different frontend with limited features).

This is really very confusing...
Titel: Re: Colours disappearing from subtitles
Beitrag von: jaydear am September 08, 2018, 21:16:23
No fuzz! Orderly, precise & efficient 8)
1. TSD - EPG, fix TS
2. VRD - frame accurate edit TS
3. VidCoder/Handbrake - compress to mp4
4. MKVToolNix - mux TS subs & mp4 to mkv

Step 4 added recently. Doesn't affect previous order, easy & fast.
Bug: monochrome subs.
Result: 95/100 ain't bad :)

Mam, you have confused VRD and Vidcoder ;D
I love HB, but it needs VidCoder to drive it to excellence.
Titel: Re: Colours disappearing from subtitles
Beitrag von: Mam am September 08, 2018, 21:22:07
Zitat von: jaydear am September 08, 2018, 21:16:23
No fuzz! Orderly, precise & efficient 8)
1. TSD - EPG, fix TS
Hmm, if you change this step SLIGHTLY only, you will get the colors that you want  ;D
Just do an "approximate" cutting (+/- a few seconds) and let TSD extract the subs from the teletext too.
(if you want it really accurate, cut @-5s and tell TSD in the config that it should add 5s to the base of each generated sub)

Then you will have "good" subs that you can later on use in step #4 and mux them back in.
Titel: Re: Colours disappearing from subtitles
Beitrag von: jaydear am September 08, 2018, 22:00:07
Zitat von: Mam am September 08, 2018, 21:22:07
... if you change this step SLIGHTLY only, you will get the colors that you want  ;D

Initially, I did try something along those lines, but of course when I did my editing in VRD to remove unwanted bits, each approx. 4 minutes long, the files no longer matched :(
Titel: Re: Colours disappearing from subtitles
Beitrag von: jaydear am September 08, 2018, 22:07:02
Zitat von: AX98 am September 08, 2018, 21:44:57
Try the free DivX Converter.

Thanks, I'll check it out :)
Titel: Re: Colours disappearing from subtitles
Beitrag von: jaydear am September 09, 2018, 01:49:13
Same result as VidCoder/Handbrake: DivX Converter can't see the subs :(
Titel: Re: Colours disappearing from subtitles
Beitrag von: Cypheros am September 09, 2018, 12:53:31
A tested, simple and working solution to create a MKV with SRT subtitles including color information. Source of the subtitles can be DVB or Teletext.
Used tools: TS-Doctor 2.1 & MKVToolnix 26.0

Sample file is "ABC2_cutted.ts", a file from Australien TV jaydear was so nice to provide two years ago.

To have Teletext and DVB subtitles converted to SRT, the following options under Settings/Preferences/"Teletext/Subtitles" have to be activated:

It's recommended to remove teletext and DVB subtitle tracks from the ts stream as they are redundant.

If I open "ABC2_cutted.ts", uncheck Teletext and click on "Save new file", TS-Doctor will create a file named Doctor Who.ts and a file named Doctor Who.eng.srt.

Now I open mkvtoolnix-gui.exe (can also be found in TS-Doctor under Tools/Expert Tools) and add the two files as "source files" (see attached image).

"Start multiplexing" will create a MKV file with colorful subtitles (see second image).
Titel: Re: Colours disappearing from subtitles
Beitrag von: Mam am September 09, 2018, 17:54:05
Zitat von: jaydear am September 08, 2018, 22:00:07
Initially, I did try something along those lines, but of course when I did my editing in VRD to remove unwanted bits, each approx. 4 minutes long, the files no longer matched :(
That's why I suggested a "rough cutting" with TSD. It might not be frame accurate (though for me its absolutely accurate enough?!?!?!?) but you can get pretty close to 1-3 seconds. The resulting subs will be close to that, and if you use the correction value within TSD, they will be totally matching.


Or are you cutting off parts from the MIDDLE of the film???
Titel: Re: Colours disappearing from subtitles
Beitrag von: jaydear am September 09, 2018, 20:51:14
Zitat von: Mam am September 09, 2018, 17:54:05
Or are you cutting off parts from the MIDDLE of the film???

Correct... Source is off-air, not from disc.
Titel: Re: Colours disappearing from subtitles
Beitrag von: jaydear am September 09, 2018, 20:57:39
Zitat von: Cypheros am September 09, 2018, 12:53:31
A tested, simple and working solution to create a MKV with SRT subtitles including color information.

Many thanks for that detailed guide. I'll test it today :D  ABC may not be the best test though... no ads to remove from middle :-[
Titel: Re: Colours disappearing from subtitles
Beitrag von: jaydear am September 10, 2018, 02:06:30
The only subs file I get from TSD when set up as you described is a "filename.[801].srt". I think that means there's only a teletext stream in the original TS file, i.e. no DVB subs (whatever they are). Anyway, the problem remains that editing in VRD AFTER TSD has done it's work leaves gaps and makes the A+V and Subs files different durations.

Stupid question time:
Here's a sample extract of a "filename.[801].srt" from TSD, read in Subtitle Edit-
3
00:28:01,740 --> 00:28:03,740
<font color="yellow">I love you.</font>
<font color="yellow">Please don't go.</font>

4
00:28:03,800 --> 00:28:05,620
(SAD GUITAR MUSIC)

5
00:28:05,680 --> 00:28:07,100
<font color="green">(CAR'S DOOR CLOSES)</font>

6
00:28:08,460 --> 00:28:10,900
<font color="green">(DOORBELL)</font>
<font color="white">SONG: # Wahh-ahhh</font>

7
00:28:12,480 --> 00:28:14,940
# Wahh-ahhh-ahhh... #

8
00:28:15,000 --> 00:28:17,580
<font color="yellow">Sandy, call me.</font>

9
00:28:17,640 --> 00:28:18,740
<font color="green">SANDY: OK.</font>

10
00:28:18,800 --> 00:28:20,660
<font color="cyan">ALL: Hey!</font>

What is it in that simple text that mkvtoolnix won't read when it's a separate stream within a TS file? To me it seems that anything enclosed in "< >" brackets is intentionally stripped out. I just don't get why.

I also don't get why teletext is so gross and ugly when viewed live (at least on my TV's). It looks like Commodore 64 text.

Where does OCR come into all this? If the text is burnt into the video the only need for OCR I can see is to get a script for some purpose which evades me just now.

There's so much I don't know about subs! Sorry for the stupid questions.
Titel: Re: Colours disappearing from subtitles
Beitrag von: Mam am September 10, 2018, 07:41:01
Zitat von: jaydear am September 10, 2018, 02:06:30
Anyway, the problem remains that editing in VRD AFTER TSD has done it's work leaves gaps and makes the A+V and Subs files different durations.
Which is perfectly logical considering your "workflow". TSD cannot know that you later on cut out parts from the middle, so it cannot calculate the resulting jump in time and modify the later subs accordingly.
Cutting with TSD is not frameaccurate, but not THAT bad too. So why dont you try my idea of an at least "rough" cut (+/- ~<1 second) with TSD and if you dont like the junctions later on, you can still use your old tool and cut out the disturbing frames. This would result in a "very close to reality" sub files (of course, if you cut out many parts, the errors in timings will add up). But I think, mostly you wont need any re-cutting, and even IF, there will only be a few frames affected so that the total difference in time will be almost invisible.
Its just a matter of good will and some training  ;D
Titel: Re: Colours disappearing from subtitles
Beitrag von: jaydear am September 10, 2018, 09:34:18
Zitat von: Mam am September 10, 2018, 07:41:01
...why dont you try my idea of an at least "rough" cut (+/- ~<1 second) with TSD...
OK, I'll give it another try (last time was in v1.2x) :)  I don't know if I frames are used more 'economically' here, but the result back then was visually jarring.