IPv6 experiments / lessons learned

During the last couple of days I did some experiments with IPv6 connectivity / applications / configuration.
For nearly two years I already got two sixxs.net tunnels. One for a server and one for my home connectivity.
I never got aiccu working on Mac OSX so the home tunnel was down most of the time.

Finally it got to me and I worked on getting 2 subnets now, again, one for the home network and one for the servers.
For the gentoo servers I used the router howto from http://www.gentoo.de/doc/de/ipv6.xml with the radvd configuration.
RADVD is a router advertisement daemon for ipv6 networks. IPv6 has a mechanism for auto configuration where the router advertisement daemon sends advertisements about the supported prefix (aka network/netmask in IPv4 world) and its own ip address for the gateway. So far it seems like most ipv6 stacks have this auto configuration included by default so every IPv6 enabled server in the reachable network suddenly has a IPv6 address. I never knew that that many servers of mine are IPv6-enabled and even quite some servers of my isp were suddently connected through IPv6 (getting me a curious call of my ISP ;-)).
Thats the first thing to be worried about, suddenly they are all connected to the big bad internet without correct reverse dns entries, firewalls and the like.
Speaking of firewalls, usually you don’t have a IPv6 firewall up at this moment. Your old ipv4 firewall rules won’t catch any ipv6 traffic. Therefore, again, every IPv6 enabled host is exposed to the world without proper protection. Thats even worse if you open a tunnel to your home network as the home network is most often connected through some router doing nat and internally just using private ip addresses so that the hosts are not exposed to the outside world at all. With opening the tunnel and enabling the radvd service you got them out in the open world either.

On my home network I got a CentOS5 server running which is doing some smb service and the like.
I got that one connected to the sixxs tunnel and started the radvd service on that box. So far so good, Mac OSX has IPv6 enabled with autoconfiguration by default so. So the hosts got the IPv6 addresses and routing.
ping6 worked (btw. nice to have most tools available as ipv6 cmds with just 6 at the end) but the browser delivered no IPv6 website. There you are, CentOS5 / RHEL HAVE a ip6tables ruleset enabled by default and that one was just open for icmp (ping) messages. Good protection but cost me a while to diagnose. So I opened some more loopholes for the IPv6 connection on the home network for smtp, imap, http, https and dns and still let the radvd daemon running.
At the server network I disabled the radvd service and manually set ipv6 addresses and gateway so that I won’t disturb neighbours in the network anymore :-). A strict ip6tables ruleset was enabled too.
For fun I went through the IPv6 certification by HE.net and got as far as to prove that I got:

• ipv6 connectivity
• an ipv6 enabled webserver
• an ipv6 enabled mailaddress (yes my main mysnip.de mail address is now ipv6 enabled!)
• reverse dns entries for my ipv6 enabled hosts (powerdns has no problems with that)

The step which still gives me trouble is that I can’t give fully ipv6 enabled nameservers to the outside world. My main nameserver is ipv6 enabled but the secondary ones from inwx.de don’t have ipv6 connectivity or AAAA entries so there’s not much I can about it.
Skimming through the maillogs on my mailserver I was stunned to see that *a lot* of spam is trying to deliver through IPv6 already. postgrey is working with ipv6 without trouble, amavis / spam assassin too so there’s not really a problem. Seems like spammers adapt more quickly to the new technologies though. On the other hand I found that freenet.de (a german ISP) got its mailservers connected through IPv6 already and is publishing AAAA entries for them. Therefore some mail is already delivered through IPv6.
In the near future I might try to offer some experimental IPv6 access to the services provided but without any native ipv6 connectivity (anyone knows if TeliaSonera is offering it and if it poses additional costs?) that doesn’t make too much sense for production.

At least now I can check how the applications I’m using and providing are working with IPv6. Also Phorum needs to be checked for that.

Dell 2650/PERC 3/Di with kernel > 2.6.22 and XFS

As it took me a day to find out I wanted to post my findings here too.

I got a used Dell Poweredge 2650 and (as usual) installed Gentoo on it. First I got a faulty harddisk in the RAID5 and rebuilding took like 6 hours.
So I didn’t mind slow io performance which I accounted to the rebuild in process.
Unfortunately it still didn’t get better when the rebuild was finished. Taking seconds for a simple “ls”, installing gentoo-sources took more than a hour and the like. I did all firmware bios updates until none were available anymore. Still, no dice.
Searching around the Web I stumbled about this post and this post (from the same author) which are pointing to issues with the most recent aacraid driver but no relation to XFS yet.
Nearly convinced to downgrade the kernel or at least the aacraid driver I did a search in the gentoo forums and finally found the solution.
Mounting the XFS filesystems with nobarrier brought the speed back to normal. Personally I would have never thought of that solution but it seems like the newer aacraid doesn’t report back that write-barrier is a bad idea on the PERC 3/Di.

Now up for the task to try to get OpenManage running on gentoo … lets try if and exotic approach helps.

Nginx, finally!

Seeing the notice that the license on my Litespeed webserver is expiring again (yearly payments 😦 ) I finally started to move my sites to nginx (together with a move in datacenters so that webserver configuration was to be done anyway).
There were some more webservers in the run but I ended up with nginx.
Some others, lighttpd (got a bit silent over there and I don’t want to put my sites on a dying project), cherokee (now even with a webinterface!, but documentation is a bit sparse and the latest release seems inconsistent with the configuration – I simply couldn’t find out to do what I wanted to do) and the original Litespeed webserver.
In the end I wanted to come back to an open source webserver which doesn’t lock me in like that.
LSWS had some regressions in the last versions and one always has to wait for the developer team to fix them (even though they are quick) as no one else can dig into the code and also no one can write modules or enhancements because of the closed source.
Also there were some features which are now only available in the enterprise (aka paid) version which I don’t want to be forced to use forever. Also in the last year(s) its simply more directed to hosting companies or similar which are using native httpd.conf files and not doing the configuration in the webinterface they are offering. Some features are even only working with using httpd.conf entries.
Oh and the free version doesn’t offer x86-64 versions therefore I needed compat libs.
Therefore better do the cut now and use something else.
Nginx has the fastcgi loadbalancing I want, rewrite rules, great configurability and a very active community (and developers).
The only thing I’m really missing there is the possibility to use .htaccess files which forced me to search for the .htaccess files and turn their rules into native nginx configuration entries. Oh, one feature I forgot, reloading the configuration without doing a full restart of the webserver is neat too :).
All issues I had could be quickly solved by either searching the maillist archive or posting there.

Don’t get me wrong. I still recommend LSWS to users who want to have an easy to use webserver with great performance as a drop in replacement for Apache supporting most of the previous features out of the box but its simply not for me anymore.

laws and the use of logging IPs

in the light of recent court-decisions in germany ( german article ) which essentially disallows logging of IPs I’m wondering what one would really need it for?

I’m using IP-logging/-tracking in multiple ways:
1. statistics about visits and recurring users
2. storing it with forum-posts to allow law enforcement in case some user really goes over the line
3. tracking requests in a given time by IP to automatically block potential attacks

So what of that could be avoided?

For 1. , one could just ignore logging the ip but trying to count visits and recurring users would be impossible with that. What now? Maybe logging a md5/shaX of the ip to have some unique key per IP? Wouldn’t that still fall under the rule from the court as you could find out which was the actual IP?
Counting visits is an important tool for getting advertisers to advertise at a page (In my opinion). Any ideas?

For 2. , guess one could disable that but would I be responsible then for each and every forum-post because the real poster can’t be retrieved? (Yeah, laws in german are bad for the one offering the forum after all 😦 )
On the other hand there is the upcoming data retention ( german news collection about this topic ) which is planned for keeping all records for 6 months (!!!). So for now I should remove all tracking of ip-addresses just to be forced to store it for 6 months a while later?

For 3. , this behaviour gives me another problem too. Trying to load-balance over multiple webservers usually goes through a reverse proxy in front of the webservers which would always give the REMOTE_ADDR of the reverse-proxy to the apps. So the reverse-proxy would need to add this security layer. But I really failed to find one doing this up to know.
But is that really needed and I’m just oversensitive in this area? Do I need to accept any number of requests/s from any user?

Are there other use-cases for logging IPs?

How are other users handling this?

whats up with lighttpd?

Is it just me or has development on lighttpd slowed down in the last months?
Last commit July 25th ( trac-timeline ) which was for a release
which opened a couple of problems ( http://www.lighttpd.net/2007/7/24/1-4-16-let-s-ship-it, blog-entry ) and none of them are fixed until now.
Also there are tickets which are (at least for me) showstopper bugs like the mod_extforward breaking url-matches which is open for 3 months now and got no comment by a developer.
When I read the page from the original author I see lots of comments about mysql-proxy and I just hope that its a temporary thing instead of him switching his attention completely to something different.
On the other hand – modlogan died silently when Jan started developing lighttpd :(.

Lighttpd got nice attention as is even mentioned by netcraft in their webserver-statistics but it should still be actively maintained and all open bugs which are not feature enhancements should be fixed as soon as possible – if there are enough developers on the project left to do this.

The new db-layer in Phorum-5.2 kicks ass

Thanks for maurice’s pulling me into the db-layer I added my “own” mysql-mysnip-layer which handles the queries done.
The change from 5.1 to 5.2 brought up a split db-layer with one file containing all the queries for mysql and calling a function for actually running the query and (optionally)
returning the rows.
And that second part is simply extension specific like for mysql and mysqli extensions. I think it would be simple to add another for pdo but thats a different topic.
For now I added another “extension”-specific part for mysnip which looks into the queries to check if they need rewriting for the partitions used.
The partition-specific tables have some marker in there which tells where the partition number should be and that marker is replaced with the partition-id on querying.
If the partition-id is not yet available in for the current forum-id its retrieved from the database.

That functionality allows me finally to run through all the upgrades as we’ve added a flag to the queries which need to be run for each partition.
We stumbled about it when I was wondering how real-name upgrades are handled now as these have to run through all partitions where that user could have been posting to.
The upgrades from my early version to current 5.2 take a while but overall they are running fine.
Another nice thing is that I don’t have to hack the actual layer which contains the queries and can take that part from the distributed code, only the “extension-specific” code changed and is in a separate file. No more relying on a module to run and rewrite the table-names which would only run once on a page-view.

To cut a long story short: its great and should allow you all kinds of changes to the db-layers without touching the queries. It should even be much easier to write a layer for another database-system (anyone volunteering? ;)).

building a HA/LB solution

I’m currently in the process of trying to build a HA/LB solution for my forums.
Currently HA (HighAvailability) is created by running heartbeat on the two webserver-“nodes” with automatic ip-takeover and a mysql-slave which gets all the data from the main-db-server (but needs manual takeover).
LB (LoadBalancing) is done with FastCGI-Loadbalancing in LiteSpeed-Webserver but I’m not satisfied with the results as it seems that the first host is getting much more load than the second one.

Therefore I played with some Virtual Machines, one running haproxy ( http://haproxy.1wt.eu/ ), two running lighttpd with fcgi-php.
So far it worked good but taking down one of the webservers still gave me some failed requests if it was running under “siege”. Thats something I wanted to avoid.
Lighttpd was simply choosen because of mod_extforward so that I could keep the original hosts ip in the REMOTE_ADDR and its support for fcgi-php.

But as I wrote in an earlier post there is one feature I’d badly miss in lighttpd and which really keeps me from switching:
.htaccess-support or generally spoken: dynamic configuration changes without changing the main-configuration and the need for a webserver reload.
I found one thread in the lighttpd-forums which sounds promising.
Reading dynamic configuration from mysql is something I’d love to see. It would kick ass :).
Yeah, sure. Lighttpd would have to work without mysql-connection too, some fallback mechanism needs to be in place but that would solve at least most of my problems.
For my own DoS functionality I need a way to block connections on the webserver-level before it even reaches PHP.

So there are some problems or lets better call it “tasks” left to solve for my HA/LB solution:
– find the right webserver to implement that
– build a solution to merge the logs and process them for statistics
– find out how to get haproxy (or another loadbalancing solution) to send failed requests to another backend in case of one going down

And the big task:
– find some automatic solution for mysql-takeover (without DRBD, which I don’t trust because of its network-based nature ;))
Any ideas anyone?

To lighttpd or not to lighttpd

So for 3 months lighttpd is now in the top 5 list of netcraft statistics.
I actually tried lighttpd before using LiteSpeed-Webserver which is a commercial product (with a free standard-version) but for my use-case superior to use. Maybe they are on par performance-wise, I don’t know and didn’t do enough benchmarks to tell but the usability is totally different.
According to netcraft there are more than a million domains hosted on lighttpd now but why is there no Webinterface to configure it? Do the users see this as useless? I don’t really like to be depending on SSH-access for changing something in my webserver configuration when I’m on the road and missing input validation like a webinterface could do.
Also why is there no support to use .htaccess-files or at least search for .htaccess-files and convert them to something lighttpd likes? LiteSpeed supports .htaccess files with a cache so that it isn’t as much a performance hit as it was previously.
I would be really afraid of opening lots of holes while switching to lighttpd because I secured a ton of directories with simply a “Deny from all” in .htaccess-files and sometimes “Basic Authentication”.
Why does it have to be so hard? 😉

Charset hell (and mysql-upgrades)

Oh yeah, some people will remember me talking about charset hell when it wasn’t really that bad (yet).
But in the near future I will surely have to solve some charset problems for MySnip.de.
It is still running MySQL-4.0.x which didn’t support charsets like MySQL-4.1 did and even more MySQL-5 is doing now.
But as MySQL-4.0 support has run out I really need to upgrade soon and then I’ll probably find myself in a lot of charset troubles ;).
First one will start with the upgrade itself. How will it look like after the upgrade? Default charset will probably be latin1 so it *should* be fine. But will it?
Beside that this upgrade is already some trouble as I will need to dump/restore the whole databases which are quite some GB’s and I hate to take the services down for some hours.

For now I’m finding enough excuses like I need to wait for Phorum-5.2 until I can upgrade but in the end I can’t do it all at once.
I still hope that Phorum-5.2 will still run on MySQL4 so that I can upgrade Phorum first and then convert to MySQL5 (for now I see no problems with that).
But Brian has announced that he doesn’t care about PHP4 and MySQL4 either as we are all developing on MySQL5 and PHP5.2 (which is correct, my development environment is like that too) – we’ll see what I can do about it ;).

Seems like I’m running a really explosive mixture with PHP4.4.x, MySQL4.0.x on my production boxes and doing development mostly on PHP-5.2.x and MySQL5.0.x.
Fortunately it seems like Phorum itself is not as vulnerable to changes in PHP itself as other apps because of its NON-OO-code and up to now we had always MySQL4 in mind as it was the requirement for Phorum-5.1 which is the current stable version.

Somewhen I’ll run into walls, thats for sure …

So that is it …

… another blog from one of these web-guys.
Whats a web-guy? I see it more as developers active in the web-community or something like that or do you think thats something else?

Actually it was Brian who brought me to blogging and wordpress alltogether and therefore that title … ;).

At first, let me introduce myself:
Thomas Seifert from Berlin, Germany.
One of the three main-developers of Phorum.
My own project / “company” is MySnip.de which is also the cause why I ended up as Phorum-Developer as I’m using Phorum as the base application for the forum-hosting done there.
Other projects? Hmm, fotoii.com is one of them but there isn’t much traffic yet.
As you can see, all my web-projects are currently build with on PHP and MySQL and usually on Linux.
I know, there are a lot of other combinations possible but PHP/MySQL is IMHO the best combination ever.
You don’t have to worry about licensing costs when you start a project (just had that problem with a work project I’m involved in) and PHP allows for really rapid development and not coding weeks before seeing any result. MySQL is another problem though. Yeah, its fast and lean but it has changed much over the last couple of years (more about thatin another blog post later).