Encoding for Korean radio streams

Submitted by Minsu on Fri, 2010-08-06 22:40

Forums:

Hi all!

I'm just discovering listening music files and radio on console, and mocp ist really great! I've got a Linux-UTF-8-fs and showing Korean characters for Korean music files works fine. Now I want to go a step further and listen to Korean radio streams, like say$ mocp http://www.mukulcast.comBut while the streaming works fine, the played titles don't show up when it comes to Korean characters (just mostly spaces and sometimes cruel symbols). Is there a way to fix this?

The stream is encoded in a local Korean encoding (and not in UTF-8) I guess. I played around with some options in the config-file. I tried eg ko_KR.utf8, ko_KR.euc-kr, WINDOWS-949, ISO--2022-KR, "KS X 1001" (see also http://en.wikipedia.org/wiki/Character_encoding a.t.l.) and options like ID3v1TagsEncoding =, FileNamesIconv =, UseRCC = yes but no dice. If moc doesn't crash, it just ignores the locale:Option 'FileNamesIconv' was ignored, remove it from the configuration file.Maybe moc-config uses other conventions?

I use moc 2.4.4-2 (Arch Linux), and it is linked to libncursesw:

$ ldd /usr/bin/mocp
linux-gate.so.1 => (0xb77f0000)
libasound.so.2 => /usr/lib/libasound.so.2 (0xb76ff000)
libjack.so.0 => /usr/lib/libjack.so.0 (0xb76e7000)
libpthread.so.0 => /lib/libpthread.so.0 (0xb76cd000)
libsamplerate.so.0 => /usr/lib/libsamplerate.so.0 (0xb7562000)
libcurl.so.4 => /usr/lib/libcurl.so.4 (0xb7514000)
libncursesw.so.5 => /lib/libncursesw.so.5 (0xb74c9000)
libltdl.so.7 => /usr/lib/libltdl.so.7 (0xb74c1000)
libm.so.6 => /lib/libm.so.6 (0xb749c000)
libc.so.6 => /lib/libc.so.6 (0xb7351000)
libdl.so.2 => /lib/libdl.so.2 (0xb734d000)
librt.so.1 => /lib/librt.so.1 (0xb7343000)
/lib/ld-linux.so.2 (0xb77f1000)
libssl.so.1.0.0 => /usr/lib/libssl.so.1.0.0 (0xb72f2000)
libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 (0xb7178000)
libz.so.1 => /usr/lib/libz.so.1 (0xb7163000)

Any help is appreciated.
Thanks
Minsu

lenoch

Sat, 2010-08-07 13:06

Hi,

I had a brief look at the code and I guess that metadata coming from internet streams don't get their encoding converted at all. The encoding is EUC-KR, by the way.
I hacked together some code and as it seems, I'll introduce a new setting called StreamMetadataEncoding.
Hopefully, I'll publish the patch soon, the proof-of-concept code needs some polishing :-)

Minsu

Sat, 2010-08-07 17:11

Thank you very much! I'm looking forward to it :-)
(...and sorry for my typos.)

Will there be an updated package ('plain' or for svn)?
Well, i'm a noob and don't know how to patch at all~ :-P

Edit: Ah, one more thing. If moc can't detect the stream's locale by itself (yet), what about a special commandline-option? One can make easily aliases for listening streams with different Asian or whatever locales then. -- If I didn't miss s.th., an option like that is not yet implemented.

lenoch

Sat, 2010-08-07 18:20

Being an Arch user as well, I can customize a PKGBUILD (probably this one: http://aur.archlinux.org/packages.php?ID=12793) to contain the patch, just for an excercise :-)

If you will need to choose between multiple encodings, I advise you to make configuration files with the soon-to-be StreamMetadataEncoding option set differently, e.g. moc-config-EUC-KR, moc-config-cp949 and so on. MOC already supports loading separate configs via -C (or --config) command-line option. (Or even -M or --moc-dir for a whole configuration directory.) I think that you will have to stop the MOC server process (by pressing Shift-Q) to be able to use new settings.

Stay tuned! In a couple hours, I hope to get this done :-)

lenoch

Sun, 2010-08-08 02:40

Well, here we go.

The new configuration option is functional and documented in config.example.
I prepared an archive containing a modified PKGBUILD and my patch, but I have the files separately, too.

You can get it all here:
http://www.fi.muni.cz/~xsvobo15/moc-svn-sme.tar.bz2
http://www.fi.muni.cz/~xsvobo15/PKGBUILD
http://www.fi.muni.cz/~xsvobo15/moc-r2246-stream-metadata-encoding.patch

Run "makepkg" in a directory where you unpacked the archive (or, equally, where you put those two files). Install the resulting package with "pacman -U moc-svn-sme-2246-1-i686.pkg.tar.xz"

Hope you find it useful :-)

Edit: If you use i686, I can offer a package, too:
http://www.fi.muni.cz/~xsvobo15/moc-svn-sme-2246-1-i686.pkg.tar.xz

Minsu

Sun, 2010-08-08 22:15

Hi lenoch,

thx very much for your pkg. :) I just unistalled the regular moc and tried it. However, I can't start moc:
<br /> $ mocp http://www.mukulcast.com</p> <p>FATAL_ERROR: Error in config file, line 135.<br />

And it's the same when starting it without argument, using your pre-built i686-pkg or building it myself via PKGBUILD.

lenoch

Sun, 2010-08-08 22:27

As MOC says, there's something wrong in your configuration file. Can you post it (at least the neighbourhood of line 135)?

Minsu

Mon, 2010-08-09 04:56

Sorry, it was just too late! :P

After correcting the config, it starts of course.
Well, the titles' display changed somehow, but ntl it is not as it should be. Instead of (orthographically correct) Hangul (the Korean script) MOC now shows mostly "###" like > #귣###### (Brand New Day) - #츸## or > ##ȿ## - U-Go-Girl (With. ####), no matter what configure for ID3v1TagsEncoding or UseRCC. -- Ehm, did you compile your pkg just for EUC-KR then?

On the other side, when firing up a terminal with Korean locale, the titles are displayed properly, like:
<br /> $ LANG=ko_KR.euc-kr terminal &amp;<br /> $ mocp http://www.mukulcast.com<br /> ...<br /> &gt; SG 워너비 - 사랑가<br />

Maybe this method fits also other other locales and the standard MOC-package as well.
Will check this out later; for now, I'm off for a couple of hours.

CU

lenoch

Mon, 2010-08-09 09:34

Well all you had to do was:
- just set the new StreamMetadataEncoding option properly (to EUC-KR):

...
# Use librcc to filenames and directory names encoding.
#UseRCCForFilesystem = yes

# Convert a stream title from the following encoding.
# If blank, this option does nothing (iconv's behaviour).
StreamMetadataEncoding = EUC-KR

# Remember the playlist after exit?
SavePlaylist = yes
...

- run MOC in its native UTF-8 mode (don't change your $LANG; it has to be ko_KR.UTF-8)
- and there you go...

It looks like this on my terminal:
├────┤Playing... ├──────────┤ Headphone 74% ├───┤>000:00:00├┤
│ > 럼블피쉬 (Rumble Fish) - 그대 내게 다시 │
│05:56 [00:00] 44kHz 128kbps [STEREO] [NET] [SHUFFLE] [REPEAT] [NEXT] │
└┤ ├┘

Even though I don't have a font set properly, CJK characters show as double-width and copying them e.g. into Firefox proves that they are indeed there.

Edit:
By the way, I tried running:
LANG=ko_KR.UTF-8 mocp http://www.mukulcast.com/
And with locales not in place, I was given ### characters in the terminal (the cs_CZ.UTF-8 XTerm title was OK though). So you have to ensure that you have the proper line uncommented in /etc/locale.gen:

...
#ko_KR.EUC-KR EUC-KR # probably uncommented too
ko_KR.UTF-8 UTF-8 # uncommented!
#kok_IN UTF-8
...

Then you have to run locale-gen as root. This made Korean characters appear for me - because I didn't have the locale.

Minsu

Mon, 2010-08-09 19:33


Well all you had to do was:
- just set the new StreamMetadataEncoding option properly (to EUC-KR):

...
# Use librcc to filenames and directory names encoding.
#UseRCCForFilesystem = yes

# Convert a stream title from the following encoding.
# If blank, this option does nothing (iconv's behaviour).
StreamMetadataEncoding = EUC-KR
...


I see.
So, getting foreign radio stream playlists properly displayed means creating several config files and loading one with -C [i]config-file[/i] on the command line. Note that the argument is not case-sensitive, either EUC-KR or euc-kr match. -- Well, maybe someday it'd be possible to pass StreamMetadataEncoding directly as an option to mocp instead of shuffling config files (there are plenty of local Chinese locales, for example). But at first, creating StreamMetadataEncoding-variable was very good idea, thanks very much, lenoch! :D
You should consider to publish an official update including this feature soon.

Well, if you're interested in testing other locales of international radio streams, you may check e.g. the addresses shown here: (1) http://www.omniglot.com/links/radio.htm (2) http://www.omniglot.com/links/radio2.htm (Ntl the lists seem to be a bit outdated.)

Additional note for multiple-byte-encoding plain console users: invoke a framebuffer-terminal first, eg with
<br /> $ fbterm 2&gt;/dev/null<br />


Edit:
By the way, I tried running:
LANG=ko_KR.UTF-8 mocp http://www.mukulcast.com/
And with locales not in place, I was given ### characters in the terminal ...

On my system, all Korean and some more locales were already activated.

Cheers
Minsu

PS:
[offtopic]
Am I right, that MOC doesn't support .m3u-, .asx-, .pls-, and mms://-streams yet? If yes, how, and if not, are there plans that MOC would support these formats some day?
[/offtopic]

lenoch

Mon, 2010-08-09 20:51

Oops, it looks like the option either has to be set (i.e. not blank) or commented out, sorry for the error :-)
Fixed files are available in the mentioned location to make it easier for other adopters of the patch. (I also made it possible to rebuild the package easier.)

Although I usually don't listen to radio, I can recall having to wget the "playlist" first and find the correct URL in it. Implementing this in MOC is possible, it seems like more work though ;-)

As for "DJ'ing" with config files, there exists a simple solution - to script it. It could look like this:
radio http://www.mukulcast.com/ EUC-KR
The command (written e.g. in Bash) would take your current config, use e.g. sed to set the StreamMetadataEncoding option according to the (optional?) command's parameter and finally rerun MOC (the supplied config is ignored if the server is already running). That would be my solution - if you'd like it, I can try to implement it :-)

Lastly, as I'm not MOC's maintainer, I can only offer an unofficial package. I'm planning to collect useful/interesting patches from this forum and use them in a new PKGBUILD that would be available in AUR. (Of course a single patch will be available for other distro users.)

Minsu1

Wed, 2010-08-11 21:25

Scripting sounds good, but because I haven't needed any own shell- or other scripts yet, I'm not experienced with this at all -- so you are the real geek here, lenoch ;)
Well, some day I'll learn at least how to write cool shell scripts, so far I only know that they should start with a shebang or so, and one must make them executable with chmod a+x.

Maybe a shell script could solve both problems with the locales and providing a workaround for .pls- and .m3u-streams. A rough concept for the second item could be:
* using wget for saving the file contents of a full .pls-/.m3u-stream url
* using sed(?) for scanning the file for the necessary http-address, and using a pipe | and (s.th. like) grep for extracting and passing it to MOC
* using rm for deleting the downloaded file (for minimizing trash-files and avoiding possible renaming of following downloaded files like listen.pls.1 a.t.l.)
* using other functions for starting MOC displaying the stream's playlist properly
-- just an idea. I'm not a programmer at all :P

All what I knew was, that one can find out the needed http-url form .pls- and .m3u-streams when opening them in another browser tab or showing the contents with cat; all urls shown there will match. However, some streams need an additional port while others don't. If you like, you can check these:
http://www.wdr.de/wdrlive/media/wdr2.m3u (German)
http://www.wooriaru.com:8000/listen.pls (Korean)
(NB: mpg123 [hence its name] uses .m3u-streams directly via $ mpg123 -@ http://www.wdr.de/wdrlive/media/wdr2.m3u. Maybe there's some 'adoptable' source code in it.)


Lastly, as I'm not MOC's maintainer, I can only offer an unofficial package. I'm planning to collect useful/interesting patches from this forum and use them in a new PKGBUILD that would be available in AUR. (Of course a single patch will be available for other distro users.)

Welcome. Then you're probably 'just' an ambitious member of the community ;)

PS: unbelievable, but my posts were marked as sp*m and refused! So I (Minsu) had to create a new account.

fatblueduck

Wed, 2010-08-11 07:34

It would be a good thing if MOC had someone pushing it towards a new release.

Daper has communicated that he will not actively develop MOC.

Will MOC fall into a permanently un-maintained state? Should potential contributors shift their attention towards CMUS instead?