FTP internals [Active Mode vs Passive Mode]

In the integration process, data may needs to be fetched from the FTP and in turn has to be published in the web or published in the ESB. I think knowing some internals about the protocol is required to do that. So here I am sharing the knowledge I have on FTP protocol.

FTP is acronym for File Transfer Protocol. It basically used to transfer the file between the FTP server and the FTP client based on the user privilege. By default ftp service is accessible on the port 21. Actually port 21 is called as control port. As per the initial definition of FTP, port 20 act as data port.

Now as per RFC959, either PORT or PASV command decides the data port. FTP is stateful protocol, means it keeps the state for a transaction at all the time. You can read more about the FTP in RFC959.

Coming to the point, to configure FTP connection as integration engineer, you must know, in what mode the FTP server is working. Because there are two different modes available in FTP, Active and Passive. In all the integration tools you can see these two modes as part of FTP connection palette, you have to choose either one. Some FTP server may work in both the modes, but some may not. So it is mandatory to know which one you supposed to choose. So let me explain what does that two modes mean and how does they differ from each other. Basically in the Active Mode, we can decide on which port the FTP server should send the file. But in the Passive Mode, FTP Server decides in which port the file going to be sent out.

Active Mode [PORT command]

Connect to:
(08/24/2007 1:02:30 PM)
hostname=10.117.34.151
username=anand
startdir=
220 (vsFTPd 2.0.4)
USER anand
331 Please specify the password.
PASS ***********
230 Login successful.
SYST
215 UNIX Type: L8
FEAT
211-Features: EPRT EPSV MDTM PASV REST STREAM SIZE TVFS
211 End Connect ok!
PWD
257 "/home/anand" Get directory
TYPE A
200 Switching to ASCII mode.
PORT 10,117,34,225,5,190
200 PORT command successful. Consider using PASV.
LIST
150 Here comes the directory listing. Download Waiting for server...
226 Directory send OK
TYPE I
200 Switching to Binary mode.
PORT 10,117,34,225,7,31
200 PORT command successful.
Consider using PASV.
RETR CONFIG.SYS
150 Opening BINARY mode data connection for CONFIG.SYS (0 bytes). Download Waiting for server...
226 File send OK.

In the above FTP transaction all RED bolded lines are ftp commands. You can identify PORT commands in the transaction. PORT command accepts 6 octets delimited by comma. The first four represents ip address of the FTP server. Sometime it could be NAT address. The 5&6 octets represent 16-bit port number. Fifth octet contains higher order 8 bit address. Sixth octet contains lower order 8 bit address. So as per our example, LIST command output streamed in the port (5*256)+190=1470. And the download CONFIG.SYS ( RETR CONFIG.SYS), streamed in the port (7*256)+31=1823. In the FTP server output, you can find, "200 PORT command successful. Consider using PASV.", why because sometime requested port might not be available, so server says to consider PASV, that is nothing but Passive mode.

Passive Mode [PASV command]

TYPE I
200 Switching to Binary mode.
PASV
227 Entering Passive Mode (10,117,34,151,236,100)
RETR CONFIG.SYS
150 Opening BINARY mode data connection for CONFIG.SYS (0 bytes). Download Waiting for server...
226 File send OK. Copied (08/24/2007 3:22:08 PM): ftp://10.117.34.151/home/anand/CONFIG.SYS -> c:\apps\CONFIG.SYS

The difference in this is, as you see visibly, server decides and says the port number, "227 Entering Passive Mode (10,117,34,151,236,100)". So CONFIG.SYS will be streamed in the port (236*256)+100=60516. Here the advantage is, as server decides the port number, the administrator can give the passive port number range in ftp server settings, and can exclude and open up those ports in firewall and can NAT the firewall with FTP server. So it could be concluded that it's better to choose Passive Mode, unless client says to use Active Mode. I hope this helps all of us to better understand Active and Passive Mode of FTP.

--
Anand
anand.sadasivam@googlemail.com

SVN Synchronization - How to create mirror repository in Subversion

Hi All,

All of you know subversion is the popular version control system, that suits for a enterprise level development, as it updates revision count for the entire project (i.e., repository) on every commit, unlike CVS, as it keeps individual revision count for each file. So by revision number we can take the old codebase for the entire project on just one click or by single command.

I was trying to create mirror site for main subversion site. I thought it might be helpful for all of us whoever trying the same, So here I am sharing my experience I had on configuring Subversion for Synchronization.

Basically the utility "svnsync" is used to create the mirror repository, and it comes as part of subversion. If you download something from the source-forge site, you will be having the option to download the selected utility either from main server or from any of the mirror site. Most commonly all the major linux distribution, for example debian can be found on so many mirror sites other than the main server.

So what I was trying to achieve was,
-> Create mirror repository
-> Do the periodic sync between the main repository and the mirror repository

Let me explain the operation I tried,


----------------------------------------
Assuming there is svn repository 'main_repo' and the development goes on that. To create the mirror repository in some other system say 'mirror_repo', we need to do the following steps

step1: create a blank repository at the mirror end.
$svnadmin create mirror_repo

NOTE:Repository should be blank at the mirror end. Otherwise sync will not work


step2: create the pre-revprop-change hook
$touch mirror_repo\hook\pre-revprop-change
$cat > mirror_repo\hook\pre-revprop-change
#!/bin/sh
^d
$chmod u+x mirror_repo\hook\pre-revprop-change


step3: initialize synchronization.
$svnsync init --username 'sync_user' file:///path_to/mirror_repo http://site_name/svn_parent_dir/main_repo
Copied properties for revision 0

NOTE:sync_user is the user available in main as well as mirror repository with 'rw' permission.


step4: create cron job to get synchronization done as it should happen periodically. This sync operation replays the commit happened in the main repository to the mirror repository.
$svnsync sync file:///path_to/mirror_repo
Committed revision 1.
Copied properties for revision 1.
Committed revision 2.
Copied properties for revision 2.
Committed revision 3.
Copied properties for revision 3.
...

NOTE: If you break the operation in mid. The repository at the mirror end might get locked and when you do 'sync' again you may get error "Failed to get lock on destination repos". To rectify this you have to run the following command "svn propdel svn:sync-lock --revprop -r 0 ${TOREPO}"

So as per our example the command will be,
$svn propdel svn:sync-lock --revprop -r 0 file:///path_to/mirror_repo
property 'svn:sync-lock' deleted from repository revision 0
----------------------------------------



Points to be noted:

1)Both main and mirror repository can be specified by any of the supported protocol

http
https
svn
svn+ssh
file

NOTE:If you use protocol other than 'file', at the mirror end there sould be 'ra' layer (repository access layer),For this 'subversion-devel' package should be installed in the mirror end, because this package only contains "ra" layer.

2)It is not mandatory to run the 'svnsync' utility in mirror site. You can even run the utility in any system which has access to the main and mirror site repoistory.


Hope this helps...

--
You can reach me at anand.sadasivam@googlemail.com

JSON Made Simple

Hi All,

I was working with JSON format in one of the web application for sometime. Here I am sharing my experience with all of you. JSON is acronym for JavaScript Object Notation. It is simple a format presenting javascript values as array(s) or object(s).

JSON is nothing but collection javascript objects or arrays or both. The format is well described at http://json.org/. For the first sight it might look complicated, but it is very simple.

If you ask why JSON? then this could be the answer - JSON data can be easily manipulated in the Web UI using javascript. Also with AJAX (Asynchronous Javascript for XML) you can take the ability of manipulation till the Server Pages. There are lot of parser available in market, that can turn JSON data to the appropriate server side language data. You can find more parsers at http://json.org/

The other answer could be, if you want XML data that supposed to be manipulated in HTML based Web UI, the option you have is you have to parse it using javascript, it will lead to the code overhead. Otherwise you can handle it at server-side, it will make web page to refresh many times, even if you handle it with AJAX, slowliness will be there at all the time. So JSON is the best alternate. Convert XML data to JSON in server side script. Pump the json string thru' AJAX as part reponse to AJAX request to the Web UI, then you could manipulate easily with javascript. In this way your web page will look more dynamic.

Let me take you through the JSON format

-> Object starts with '{' and closed with '}'
-> Array starts with '[' and closed with ']'

Object:-
-> Object contains set of key value pairs separated by ':' and delimited by ','
example: {"name":"Anand Sadasivam","age":20 }

Array:-
-> Array contains set of values delimited by comma
example: [172,65,42,"Cricket","Chess","Foot Ball"]

Example JSON contains both array and object:
{"person":{"name":"Anand Sadasivam","age":20 },"details":[172,65,42,"Cricket","Chess","Foot Ball"]}

If you observe the above example, the first key-value pair contains object as value. Like that the second key value pair contains array as value.

So like that JSON can be nested n number of times.

'eval' statement:
eval statement can turn JSON data in string format to JSON object. For example

var data="{'person':{'name':'Anand Sadasivam','age':20 },'details':[172,65,42,'Cricket','Chess','Foot Ball']}";

var jsonData=eval('('+data+')');

In the above code data in the eval statement is parametrized by parenthesis to restrict object evaluates to value in case of single key value pair.

Hope this helps...

--
You can reach me at anand.sadasivam@googlemail.com