Thursday, October 20, 2005

Unicode conversion in Perl

Starting from Perl 5.6, Perl uses utf-8 for storing character strings internally (Just like Java, which also uses Unicode for storing character strings). To convert a string in native encoding to another encoding, we can think of the same way as what we usually do in Java (if you happens to have some experience in Java character set conversion, you will know what I am saying). e.g. If we want to convert a Big5 string to UTF-16 :


use Encode qw/encode decode/;

$big5str = get_big5_string();
$now_in_utf8 = decode("big5", $big5str);
$now_in_utf16 = encode("UTF-16", $now_in_utf8);



For extra information, if you to want to return an Excel spreadsheet containing Chinese characters from CGI, you can do this:


#!/usr/bin/perl
use Spreadsheet::WriteExcel;
use Encode qw/encode decode/;

my $big5str = get_big5_string();
my $now_in_utf8 = decode("big5", $big5str);
my $now_in_utf16 = encode("UTF-16", $now_in_utf8);

$filename ="excel_file.xls";
print "Content-type: application/vnd.ms-excel\n";
print "Content-Disposition: attachment; filename=$filename\n";
print "\n";

my $workbook = Spreadsheet::WriteExcel->new("-");
my $worksheet = $workbook->addworksheet();

$worksheet->write(1,1,"Non-unicode characters");
$worksheet->write_unicode(1, 2, $now_in_utf16);
$workbook->close();


For M$ Excel before version 2000, you will see square box before each cell, this is due to the BOM character generated during the encode process. Excel 2003 and OpenOffice can recognise this character correctly. To remedy the square bracket, you can just remove it:


my $now_in_utf8 = decode("big5", $big5str);
my $now_in_utf16 = encode("UTF-16", $now_in_utf8);
$now_in_utf16 =~ s/^\x{fe}\x{ff}//;

AMASS

As mentioned in Ajaxian, AMASS(Ajax MAssive Storage System) allows Ajax application to store potentially large amount of persistent data in local machine. By using a SharedObject in a hidden Flash, AMASS can store local data without user intervention for data size smaller than 100k. For larger amount of data, the system will prompt for user approval. I think this is one step closer for Ajax application to perform similar things as locally installed application.

In current stage, the main issue is it only works in Windows. It has not been tested in Linux platform, and it doesn't work in Mac :<.

Wednesday, October 19, 2005

Subversion initial setup

Subversion is a project aims to build a version control system that is a compelling replacement for CVS in the open source community. Besides usual way of doing version control, this is also a good candidate for backup of anything you can imagine. There is a free online book available, if you want to further study it.

Below are some setup notes I have jotted down during my setup. May be this will be helpful to you also.

Setup in Gentoo Linux, as apache2 module

1. If you happens to have apache 2 already installed, setting up Subversion is easy:

# emerge subversion

But in case you are using apache 1.x, you can still go ahead and let portage to install apache 2 for you. There is nothing much harm (other than use more memory) to run both apache 1.x and 2.x together. All you need is just changing the port number apache 2.x is listening to.

2. Create directory for source repository: (you can use whatever directory)

#mkdir /var/repository

3. Create user account for svn:


#mkdir /var/svn/conf
#htpasswd2 -c /var/svn/conf/svnusers newusername


The second command create a new user "newusername" and store the a/c password info in /var/svn/conf/svnusers. The -c option is for file creation, if you are adding more users, you can simply go without this option.

4. To allow apache2 to load the corresponding svn modules, modify /etc/conf.d/apache2 to add "-D DAV -D SVN", e.g.:

APACHE2_OPTS="-D SSL -D PHP4 -D FASTCGI -D DEFAULT_VHOST -D DAV -D SVN"

5. Configure the modules config file /etc/apache2/modules.d/47_mod_dav_svn.conf (you may have number other than 47, depending on the portage maintainer). Assuming we use the password file and repository directory created previously, the file may look like this:


<IfDefine SVN>
<IfModule !mod_dav_svn.c>
LoadModule dav_svn_module modules/mod_dav_svn.so
</IfModule>
<Location /repos>
DAV svn
SVNPath /file2/repos
AuthType Basic
AuthName "WINS Subversion Repository"
AuthUserFile /var/svn/conf/svnusers
Require valid-user
</Location>
<IfDefine SVN_AUTHZ>
<IfModule !mod_authz_svn.c>
LoadModule authz_svn_module modules/mod_authz_svn.so
</IfModule>
</IfDefine>
</IfDefine>


6. Create your initial repository:

#svnadmin create /var/repository

Setup in Redhat, standalone server

For illustration purpose, this time, we run subversion for standalone version, and invoked through xinetd.

1. First step is also easy:

#rpm -Uvh subversion-XXXX.rpm

Just replace XXXX with the version currently available in your system.

2. Create a user for svn in your system. For security, you should set it's login shell to nologin.

3. Add 2 lines to /etc/services: (if they don't exist yet)


subversion 3690/tcp # Subversion
subversion 3690/udp # Subversion


4. Create a file /etc/xinetd.d/svn


# subversion server
service svn
{
disable = no
socket_type = stream
protocol = tcp
wait = no
user = svnuser
server = /usr/bin/svnserve
server_args = -i
}


5. Create your initial repository:


#mkdir /var/repository
#svnadmin create /var/repository


6. During initializing the repository directory, svn creates various directories and files. In particular, there is a config file svnserve.conf inside conf directory created. There are extensive comments inside the config file, with all example setting commented out because they are already set by default by svn. Anyway, you can customize them to your needs, e.g.:


[general]
anon-access = none
auth-access = write
password-db = passwd
realm = My Subversion Realm


The above disables anonymous access, and allow write access for authenticated user. Most importantly, it defines the password file for authentication as 'passwd'. Now in the same conf directory, create this file with following content:

[users]
wins=password1
user2=password2

Using subversion
The most important command to know first is help:

$svn help

To get help on a specific topic :

$svn help update

Import initial directory for a project:

For apache module:


$svn mkdir 'http://localhost/repos/NewProject' -m 'Creating new Project'
$svn import ./MyProject 'http://localhost/repos/NewProject/trunk' -m 'Import for 1st time'


Note that '/repos' is defined in Location directive in apache config.

For standalone version, use svn:// instead.


$svn mkdir 'svn://localhost/var/repository/NewProject' -m 'Creating new Project'
$svn import ./MyProject 'svn://localhost/var/repository/NewProject/trunk' -m 'Import for 1st time


Note that the '/var/repository' are actually a full path of the directory where you create for repository.

Before performing any commit, you need to perfrom an update first :

$cd ./Myproject
$svn update


To check your checked out source status :

svn status

The svn will also mark all non-version file inside your checkout directory as '?' status.

To ignore some files from subversion status cmd (or add/import), either set it globally in ~/.subversion/config :

[miscellany]
### Set global-ignores to a set of whitespace-delimited globs
### which Subversion will ignore in its 'status' output.
global-ignores = *.o *.lo *.la #*# .*.rej *.rej .*~ *~ .#* .DS_Store *.class


or set it in svn cmd:

$cd WEB-INF
$svn propset svn:ignore "*" work/

(first arg is the wildcard pattern, second is the path)

to view back any prop set to a path:

$cd WEB-INF
$svn proplist work


To commit your changes:

$svn commit -m 'Some changes...'

Video iPod


For the past week, everyone is talking about new iPod. Newspaper said many people will die for this thing because of its low price and video capability. But to me, I think it's just an intermediate product from Apple. Given there are already few PMP supporting 640x480 screen, having good support for DivX and Xvid, I can only see the video support in iPod just as a bonus instead of a great feature for main use.

The battery leads to great disappointment too. The official spec. states the 30G version lasts for 2 hours while the 60G lasts for 3 hours in video playback. Not to mention it probably assumes a fully charged and healthy battery condition.

If video support in iPod needs to be as attractive as music support, I think at least the following features need to be found in future iPod generation:

  • Great battery life for video playback (> 10 hrs)
  • Direct support for DivX and Xvid codecs alike. rmvb support would be great too (just dreaming.)
  • Bigger screen
Bigger screen could be difficult to design, in order to retain the original iPod style and click wheel. But after I saw iRiver U10, Apple should be able to come up with a better idea on their own.

Some rumors say Apple is actively developing a Wi-Fi enabled iPod, with builtin mobile Safari for browsing and even buying tunes in ITS. No matter this rumor is true or not, but network enabled iPod is really a great feature that I personally die for long time ago. Imagine you can listen to Internet radio in public places where WiFi is available. At home, you can stream your bigger music collection from iTunes to your iPod similar to Airport Express. And best of all, we will probably have WiFi headphones to free ourselves from cable. I remember Jobs once thought Bluetooth can't deliver good quality music wirelessly to hp. But I strongly believe WiFi can make this come true. Besides, a builtin mobile Safari means a brand new portable device experience just like PSP does. Besides usual web browsing, I know PSP can now even do remote control through VNC, may be iPod can do similar thing to control your Mac remotely? OK, I know it's already too far away.

How about a Dashboard widget style of APIs for iPod? Imagine all cool things you can do with widgets now can easily ported to iPod. Hmm...

Back to real. Right now, the new Zen Vision would definitely be a better choice for PMP than iPod. Let's see which direction Apple will go in their next step in iPod.

Tuesday, October 18, 2005

Why I switch to Mac OS X

I have been a PC user for more than 15 years, and have been both Windows and Linux users since their very early version. (DOS 5.x to Windows 3.x to XP, Slackware to Redhat to Gentoo). I have also been a user of OS/2 for about 1 to 2 years.

Recently I have switched to Mac OS X for the following reasons:

  1. Easy to use, cool looking, well thought and stable GUI readily available. Supporting Chinese and those default good looking Chinese fonts just can't be available in common Linux distro by default.
  2. BSD Unix behind the scene. So I can still do whatever I can do previously with Linux (Apache, Jboss, mysql, gcc, perl,php)
  3. I am not a gamer, so most of the applications I usually use in XP do have similar equivalent in OS X.
  4. Plug & Play usage for my iPod and firewire/usb external drives. (I can even connect my plain old Creative Jukebox thr. USB, just need to install an extra app)
  5. No more virus/spamware problem.
  6. Can run some great killer applications like Photoshop/Dreamweaver etc.

For the first 2 weeks with my Mac, I am struggling with searching for similar applications I used to have in Windoz. Now my search is nearly over:

Browser -> Safari/Firefox
BT -> Azureus
FTP -> Fugu/Transmit
Text editor -> jEdit
Media Player -> VLC (and realplayer for rmvb)
chm ebooks reader -> xchm
Photo viewer -> FFview/GraphicConverter
SSH client -> Terminal!

Well, I know every platform is not for everyone. For me, I need a rock solid OS which can both do GUI and server applications. Linux should be the default answer, but the GUI from both KDE or GNOME just isn't as cool as Aqua. And you just can't plug and play with most hardware with Linux. Don't flame me on this. That's personal choice. :)

Catalyst Framework

Catalyst is a MVC(Model-View-Controller) Framework in Perl, based on the idea of Maypole and RoR. If you know Perl but want to have a taste of all the juice from RoR, but don't want to learn Ruby, Catalyst should be a good answer.

In particular, it provides URL action mappings, and CRUD application too (i.e. instant web database frontend, once you have your MODEL defined).

Thursday, June 09, 2005

Elinks

If you like lynx, you will love Elinks. It has direct mouse support, frames and tables, menus (even pop up menu), within your text terminal. Oh, did I mention it also supports tabbed browsing? (better than IE6, ha ha...)

Besides Midnight Commandar, this is another cool text (or cli) mode application with good mouse support. Anyone know any others?

Monday, June 06, 2005

Backspace + Screen + Vim + PuTTy in FreeBSD

If you use putty to ssh to FreeBSD, you will notice the backspace key won't work in vim in insert mode. The easiest solution is set the backspace to Ctrl-? in your putty setting.

Now when you starting working with gnu screen, you will met another problem. The Ctrl-? backspace now shows as ~ in your shell prompt. If you set the putty back the Ctrl-H, it will works again.

So what should we do if we want to use vim inside screen, through PuTTy in a FreeBSD environment? The solution is, use Ctrl-H in your PuTTy, and add the following code to your .vimrc file:

:if &term == "termname"
: set t_kb=^V
: fixdel
:endif

The above code is copied from vimdoc.

Sunday, June 05, 2005

GNU Screen

GNU Screen enables you to work in multiple sessions in single a telnet/ssh sesion. After you have installed it :

- run screen. Now you are presented still in your shell prompt as usual. Ctrl-a c will create another new session.
- Ctrl-a " will list all your session. Use arrow key to select one, enter to goes to that session.
- Ctrl-a A will prompt you a name for the current session.
- Ctrl-a w will shows you some info of all sessions.
- Ctrl-a Ctrl-a toggles between current and previous session.
- Ctrl-a d will detach the curren screen sessions, and goes back to your original shell prompt. In fact if you are in a remote terminal, just close your terminal emulation app will achive the same effect. After detached, screen -ls will list all the detached sessions. screen -r will restore the first one. screen -d -r will try to detach the session, then attach it. (Good for detaching the session from another remote terminal, and steal it to current one). Think about it, it's really handful!
- For help, Ctrl-a ?
- Hint: If you usually set your bash setting in .bash_profile, you can make a symbolic link of .bashrc back to this file. This way, when screen invokes bash, it will read back your profile setting.
- Hint: If you see ~ appearing when pressing backspace key, it means your terminal emulator is setting wrong. For PuTTy, change it to Ctrl-H.

Don't use vim just as vi

Most Linux nowadays comes with vim instead of vi. But many people just keep using vim as vi. (I am one of them, how about you?) Actually when I first met vim, I am excited, and try to learn all those new techniques. But later on, as using GUI editor much more often, I forgot everything. :x

Below are some difference that I think vim really stands out:

0. Visually copy/cut/paste!
- just type v to start visual mode. Then navigate to whereever you want. This will creates a marked block. Just type 'y' for yanking (copy), or 'd' for cut. Move your cursor to elsewhere, then 'p' to paste your text!
- 'V', will start marking lines
- Ctrl-v will start block mode marking!
- During cursor movement, 'o' will move to the other corner of your current cursor
- For block Visual mode, 'O' moves to the other corner, in the same line
- 'gv' highlight back the previous marked area.
- In addition to yank(y) and delete (d), you can also perform change 'c', 'I' for insert, 'A' for append. For block mode, this means you are repeating what you change for the marked areas. Pretty cool.


1. Resizable windows
- To split current session to two horizontal windows, :split
- To jump your cursor location from one to another, Ctrl-w w (or Ctrl-w Ctrl-w)
- To close one of the window created by split, just Ctrl-w c
- The usual jkhl navigation key works as expected, just add Ctrl-w before it. e.g. Ctrl-w j will move your cursor to the window below your current one.
- To create a new window of a new file, :new
- Ctrl-w + (don't just press the '=' for +, you need to add a shift key!) increase your current window by one row. Type a number before Ctrl-w will repeat this action. So 5 Ctrl-w + increases 5 rows. Similarly, Ctrl-w - will decrease by one.
- <N> Ctrl-w _ will set your current window to <N> rows. Ommitting will maximize your current window as large as possible, but still keep other windows there.

2. File browsing
- : edit <filename> (e <filename> will also do) will try open another file. You will be warned if you havent' save your current file. Also note that you can perform filename completion by pressing <Tab> key.
- Even better, for vim 6.x, edit . will brings you the directory listing. The usual vi navigation (jkhl) keys works as expected. You can perform / search. Just press enter for a directory name will navigate to this directory. Pressing enter for a file will start edit this file. 'D' (capital D) will delete the file, don't worry, it will ask you to confirm.
- You can keep editing another files during your session by using :edit , but just you need to save before moving to another. If you don't want to save yet, try :hide edit filename . (type hid instead of hide will also do.)
- Vim keeps a buffer list of all the editing sessions, and assign a number to each buffer. To view all the buffers, :ls When you knows the buffer number, you can switch to this file by :buf <N> (or even just :b <N>). Still , add hid before if you don't want to save for the moment. In the buffer list, '#' means your current session, '+' means modified session.
- Another way to switch between buffer number is <N>Ctrl-^ (press Ctrl-6), where <N> refers to the buffer number.
- To quickly leave all sessions without saving, :qa!

3. text completion
- Ah, this is really cool. Keep pressing Ctrl-N will goes through all possible strings appears in your current session.

4. Tag list
- The Taglist plugin shows all the functions/scope in another window, just like Function Lists in PsPad/Ultraedit. It will also show the buffer list at the top for easier navigation. Cooo...l!

5. Undo/Redo
- You know 'u' will do undo. For vim, you can perform multiple undo <N>u
- The other way round, Ctrl-R perform redo.

5. Insert mode change
- under insert mode, you can move around using cursor, and keep inserting text. No need to type <Esc> i

6. Marks and Jumps
- To create a mark: m[a-zA-Z] lowercase marks will lost when the current file is removed from buffer list. Upper case marks won't.
- To return to marked: either use ' or ` , follows by the marked character. e.g. 'c
- To view the mark list, :marks
- Whenever you perfrom any jumping actions (like ', /, G etc.), a new jump list is created. To view jump list, :jumps
- Use Ctrl-o for to goes up the jump list, use Ctrl-i to goes down the jump list

7. Search/Replace
- To perform global replace, with confirmation :%s/pattern/replacement/gc
where % means perform on all lines, g is multiple time in each line, c means confirm.
- just pressing * will search the word under cursor. g* search for partial match. # search in backward direction, while g# search for partial word in backward direction.

8. Command Line Editing
- To edit another file, :e , then press <TAB>, this will
show the first file in current directory. This is vim word completion. Keep pressing TAB gives you more files in the internal list in forward direction. To get back the reverse direction, Ctrl-P. Press Ctrl-D will show the whole list for overiew.
- For set option, in addition to the option keyword completion, if you TAB after =, vim will even auto complete with the current option value.
- Under command line, you can Ctrl-P to show your command history. After you press at least one time of Ctrl-P, you can use Ctrl-N to go forward the history again.
- q: will open a command line window. The usual jkhl keys works, and you can edit any line. After you have modify any available command line (or just go to the line w/o editing), press enter. Bingo, this command is executed, and the command line window is auto closed.

Misc. Hints
- To scroll your page one line at a time, use Ctrl-Y and Ctrl-E
- change your vim init setting in your ~/.vimrc
- If you use screen with vim, you can change your .bash_profile (or .bashrc) to set TERM=xterm-color. This will allow mouse=a setting to work. FYI, default screen TERM is set to 'screen'.

Remember TC++?

Remember long long time ago, DOS C/C++ programmer like to use TC++(Turboo C++)? I am one of them. :P Even nowadays when I use PsPad or Ultraedit, I still use back the plain color theme: yellow text color on blue background.

If you use putty a lot, and miss TC++, then look no further with SetEdit. Mouse support, menus, windows resize, and all those familiar TC++ interface is back to Linux.

Saturday, June 04, 2005

CoLinux with Gentoo

Do you wish to have both XP and Linux at the same time with your notebook anytime anywhere? No, I am not talking about virtual machine, which does't have good performance. Thanks to CoLinux! Now I can have the best of both world : XP for my PsPad, Filezilla, and Putty, plus good looking Chinese fonts without any needs of tweaking. And Gentoo Linux for all my web development needs!

I installed CoLinux with Gentoo inside my Libretto U100 (Japanese page). With my Libretto upgraded to 1G RAM, XP and Gentoo live happily together. Gentoo can even run without any problem after my Libretto wakes up from Standby mode. Very cool. :)

Friday, June 03, 2005

php for Perl Programmer

Just recently started to play with PHP (Little bit late, you may think,eh?). There are many similarities between Perl and PHP, but there are many differences also. This article will try to tell the main difference of PHP from Perl. I will keep updating this article while I learn more.

It doesn't mean to be a complete list, and don't ask me why not include this and that. I don't care. :P

1. Flow Control
1.1 If
More 'C'/'Java' like:

if (expression)
statement
elseif (expression)
statement
else
statement

Single statment doesn't need bracket {}.

2. Array
2.1 Hash/Array
Perl hash is now PHP array with arbitrary index. No more '@' symbol.

$array1 = array(
0 => 'value1',
'stringkey' => 'value2',
);
echo $array1[0] . ',' . $array1['stringkey'];


2.2 Append new array element
No need to use 'push'. Just assign it with blank bracket:

$array1[] = 'I will be appended at the end.';

The above only works for simple numeric indexed array. If you use the above type assignment for associative array, PHP will still add to the array, starting with index 0.

If you want to perform similar things for more than one elements, you can use array_push():

array_push($array1, "one value", "two value");


And, if you want to append an array to another array, try array_merge():

$merged_array = array_merge($array1,$array2);

Note that if duplicate keys found in both array, the latter values will overide.

2.3 Loop through array
Do like this:

foreach ($array1 as $value)
echo "value is $value<br>\n";

foreach ($array2 as $key => $value)
echo "key is $key, value is $value<br>\n";

How to create a ftp only + chroot in FreeBSD

1. If the system doesn't have an nologin alike shell, create one: (just a simple shell script)


#!/bin/sh
exit 0


save it to /usr/bin/ftponlyshell (or wherever you want)

Important! Add this file to /etc/shells

2. Create the user as usual, using the above as shell
3. Add the user to /etc/ftpchroot

you're done.

Create user in MySQL

You can easily add mysql new user as follows:

mysql -u root -p mysql
mysql> GRANT ALL PRIVILEGES ON *.* TO 'monty'@'localhost'
-> IDENTIFIED BY 'some_pass' WITH GRANT OPTION;
mysql> GRANT ALL PRIVILEGES ON *.* TO 'monty'@'%'
-> IDENTIFIED BY 'some_pass' WITH GRANT OPTION;
mysql> GRANT RELOAD,PROCESS ON *.* TO 'admin'@'localhost';
mysql> GRANT USAGE ON *.* TO 'dummy'@'localhost';
mysql> FLUSH PRIVILEGES;

Note: don't know why, the mysql won't complain if you don't supply the -p command switch, even the user really requires password. But even you have reached the mysql prompt, you won't have any further access.

By default, mysqld don't listen on external network interface other than loopback. To add it back,
comment out the following line in /etc/mysql/my.cnf (or whereever it's located)

#bind-address = 127.0.0.1

Hello.

OK. That's just my very first post.

For the moment, the purpose of this blog is trying to gather all notes for my development exploration.

See what I will post then.