Wednesday, September 23, 2009

Zip - pipeline component

Recently I got a requirement from my client to Integrate BizTalk with the Amazon cloud, Basically we have to send notification message to a FTP Location in Amazon.
As Amazon charges based upon the pay per usage, we had a requirement to compress the file before sending to the Amazon location to reduce the cost..

I tried to search for Pipeline component, Most of the pipeline component comes with the cost (
http://www.nsoftware.com/products/biztalk/default.aspx). So I decided to create my own, with the help of open source .Net Library which is available at http://www.icsharpcode.net/OpenSource/SharpZipLib/
About the component

ZipFile is a pipeline component which Archives the file in encoding stage of the send Pipeline. This is a generalized component, which can be used wherever required.
The ZipFile Name based upon the Adapter Configuration.
File Name inside the Archieve If the "OutboundTransportLocation” (
http://schemas.microsoft.com/BizTalk/2003/system-properties) has value the file name will be extracted and it will be the same name as Zip File . Else MessageID Property will be used as file name inside Zip.
The File Extension for the File inside the zip is configurable "FileExtension"
Steps

On receiving the message, the component will

  • Read the body of the message (Which is a stream) to the Zip function and the message name
  • Zip the stream, Update the Name of the Zipped File and return as a memory stream.
  • Process the memory stream to update the message body and return as message

Code snippet




public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
{
try
{
// Name of the File inside the Zip Starts Here
String MessageId ="";
String Extension = "." + _FileExtension;

MessageId = pInMsg.Context.Read("OutboundTransportLocation", "http://schemas.microsoft.com/BizTalk/2003/system-properties").ToString() ;

if (MessageId == "")
{
MessageId = pInMsg.MessageID + Extension;
}
else
{
try
{
//From OutboundTransportLocation
MessageId = MessageId.Remove(0, MessageId.LastIndexOf(@"/") + 1);
MessageId = MessageId.Remove(MessageId.LastIndexOf(@"."));
MessageId = MessageId + Extension;
}
catch
{
MessageId = pInMsg.MessageID + Extension;
}

}

// Name of the File inside the Zip Ends Here

MemoryStream outmsg = new MemoryStream();

IBaseMessagePart bodyPart = pInMsg.BodyPart;
if (bodyPart != null)
{
ZipingTheStream(bodyPart.Data, outmsg, MessageId);
}

return CreateNewMessage(pContext, pInMsg, outmsg);

}
catch (Exception ex)
{
throw new System.ApplicationException(ex.Message);
}
}

///
/// Helper function that creates a new message and copies all parts
/// and their properties. Copies new stream to current message
///

/// Context
/// Orginal BTS message
/// Contains contents of message to create
///
public static IBaseMessage CreateNewMessage(IPipelineContext pipelineContext,
IBaseMessage message, Stream streamOut)
{
if (null == pipelineContext)
throw new ArgumentNullException("pipelineContext");
if (null == message)
throw new ArgumentNullException("message");
if (null == streamOut)
throw new ArgumentNullException("streamOut");

try
{
ICloneable c = (ICloneable)message;
IBaseMessage outMsg = (IBaseMessage)c.Clone();
streamOut.Position = 0;
outMsg.BodyPart.Data = streamOut;

// Clean up resources
pipelineContext.ResourceTracker.AddResource(outMsg.BodyPart.Data);

return outMsg;
}
catch (Exception ex)
{
//EventLog.WriteEntry(ex.Source , ex.message);
throw ex;
}
}

public static void ZipingTheStream(Stream InMsgBody, Stream ZipStream, String FileName)
{
try
{
ZipOutputStream ZipOutStream = new ZipOutputStream(ZipStream);
ZipOutStream.SetLevel(9);

ZipEntry InMsgBodyEntry = new ZipEntry(ZipEntry.CleanName(FileName));
InMsgBodyEntry.DateTime = DateTime.Now;
InMsgBodyEntry.Size = InMsgBody.Length;
ZipOutStream.PutNextEntry(InMsgBodyEntry);
byte[] BufferTransfer = new byte[1024 * 1024];

int Filebyte;
Stream FileTobeZipped = InMsgBody;
for (; ; )
{
//we copy the file to the Zip stream, block after block
Filebyte = FileTobeZipped.Read(BufferTransfer, 0, 1024 * 1024);
if (Filebyte == 0) break;
ZipOutStream.Write(BufferTransfer, 0, Filebyte);
}
//ZipOutStream.

ZipOutStream.Finish();
//ZipOutStream.Close();


}
catch (Exception ex)
{
throw ex;
}
}

Friday, September 18, 2009

Apache Proxy Security Issue

Recently We were deploying a ROR (Ruby On Rails) application to be specific its Redmine. Since our web server had too many virtual host running on Apache we couldn't run the webrick web server directly on port 80. We decided to run it on Port 8000 and let apache virtual host for this redmine will be proxying to the port 8000

Whats the configutration?

-------------- Configurations Begins ---------------
<VirtualHost...... >
ProxyRequests On
ProxyVia On


....

ProxyPass / http://localhost:8000/
ProxyPassReverse / http://localhost:8000/

.....
</VirtualHost.>
-------------- Configurations Ends -----------------

Later a month we observed our web server became too slow, We saw the response taking too much time. Looking at the performance Apache was consuming more memory and cpu load.

Just a top command explained the change in apache's behaviour

Looking into Apache's access log we saw too many web requests unrelated domains to the server were accessed. Finally we realised that the apache became a proxy server and now it is acting as a proxy to many people and they access their banned sites through the apache's proxy service.

The fix is to remove the following entries

ProxyRequests On
ProxyVia On

The ROR application was still proxied because of the other entry in the VirtualHost. The application still worked and we stopped the open proxy behaviour.

Once the fix was done, we observed all the proxy request in the access logs were denied with 404 and thus the server is saved ;-)

Tuesday, September 01, 2009

SQUID Load balancing for web applications

Squid Load Balancer Configurations

A short note on we have done to make the squid load balancer working
The servers are CEnt OS 5.x

Total machines 2 (Don't ask me why it is 2, This is what I had to test and play with in my lab)
Machine 1 - IPs 192.168.5.50 and 192.168.5.51 (2 LAN cards)
Machine 2 - IPs 192.168.100 and 192.168.5.101 (2 LAN cards)

The machine 1 and 2 Will have Apache application running on them which needs to be load balanced.

Problem: We need a seperate machine which should act as load balancer for both, but we don't have any other machine than these 2.
Solution: Machine 1 will act as load balancer and also as a application server(not adviceable) but can go ahead if we run out of resource as we have no other option.

DNS Names
DOMAIN Name for public access webapp.office.lan

webapp.office.lan - 192.168.5.50
server_1_a.office.lan - 192.168.5.50
server_1_b.office.lan - 192.168.5.51

server_2_a.office.lan - 192.168.5.100
server_2_b.office.lan - 192.168.5.101

The web application is configured in both the server with apache vhosts.


The default site is webapp.office.lan and squid listens to the queries on that.

So the first part is making the application work in

server_1_b.office.lan - 192.168.5.51
server_2_b.office.lan - 192.168.5.101

Configuring apache right to make this work properly.
Since Squid and Apache are going to run on same machine(1) and are to be used in same port number 80 we need to make apache listen only for requests on IP:192.168.5.51

To do the above
Change the httpd.conf
Modify the line
Listen 80
to
Listen 192.168.5.51:80

So apache listens only the IP: 192.168.5.51 on Port 80

Now configure the vhost of the web app accordingly in Apache on IP:192.168.5.51
Verify do the site works by accessing server_1_b.office.lan

Configuring App in server_2_b.office.lan - 192.168.5.101
Since the app is now configured on a different machine we don't need to change the Apache Listen property.
but once configured just check the site works with server_2_b.office.lan URL

Squid in Action
Installing squid in CentOS is as same as installing Apache with yum installer.

Configuring Squid
Add the following lines in /etc/squid/squid.conf

------------------------------ Lines to be added in squid.conf -------------------------------
#Make SQUID Listen on PORT 80
http_port 192.168.5.50:80 defaultsite=webapp.office.lan vhost

# Mapping 192.168.5.51 as server_1
cache_peer server_1_b.office.lan parent 80 0 no-query originserver name=server_1 login=PASS
cache_peer_domain server_1 server_1_b.office.lan login=PASS

# Mapping 192.168.5.101 as server_2
cache_peer server_2_b.office.lan parent 80 0 no-query originserver name=server_2 login=PASS
cache_peer_domain server_2 server_2_b.office.lan login=PASS

cache_peer server_1_b.office.lan parent 80 0 round-robin no-query originserver login=PASS
cache_peer server_2_b.office.lan parent 80 0 round-robin no-query originserver login=PASS

----------------------------- End of lines to be added in squid.conf ----------------------


Now the squid can be restarted.
The server will be listening to webapp.office.lan each request will be diverted to different server based on the round robin flow and if any one fails the other will server the request continuously we can add 'n' servers similar to this to make the count higher
Note all the vhost in the Apache should listen to domain name "webapp.office.lan".

Hope this gave a useful information on SQUID loadbalancing, This is not only for apache but can be any webserver serving similarly.