Archive for the ‘Phorum’ Category

MAMP and virtual local hosts / IPv6

Recently I recognized that my local access to virtual hosts created with MAMP ( http://mamp.info/ ) is pretty slow. It takes a couple of seconds to actually send the request.
MAMP (Pro) adds virtual hosts to the /etc/hosts file on mac with 127.0.0.1 as its IP-address, so that I could just enter http://myapplicationname:8888 to get to the local development environment of my app.
As I got similar timeouts accessing my hosts when I enabled IPv6 for them I guessed that there was a connection.
Adding the 127.0.0.1 equivalent for IPv6 ( ::1 ) did the trick then finally.
Add
::1 myapplicationname
to the /etc/hosts and it works blazing fast as regular.

Seems like OSX nowadays always does an IPv6 lookup, even for localhosts.

Advertisements

Phorum has moved to Github

Long time, no post :).

Effective by today Phorum has moved to GitHub (https://github.com/Phorum/Core).
That means that our code repository was converted to git (I’ve had to use svn2git as the github-internal import didn’t pick up our tags and branches) and the trac tickets were imported to github issues (I wrote my own php script for that because none of the two existing scripts worked for me. Somehow they all barfed at some broken charset characters from trac or didn’t take the api request limit from github, 60 requests per minute, into account. So I wrote my own script which was taking like 3 hours to import our 900 tickets but was a breeze to implement with the well documented github api – and its php library ;-)).
The issues in github don’t support attachments yet which means that only the comments are ported over – and a link back to the trac install is provided.

One missing part is the wiki import but I’ll see what I can use of the old data.
Also our release scripts aren’t changed yet but I’ve got to see if they might just stuff the generated docs into the tagged release as github creates tarballs / zip archives from the existing tags already. Now we’d just need to download and put them on phorum.org too ;-).

It took me a while to get my local development environment to work with git but a first commit to our 5.2 branch is done now, so that actually works. Working with branches in git made my head hurt but as long as works I don’t really mind. I’m currently evaluating phpStorm for my development work which has internal github and git support and seems to handle that in an accessible way without having me to look into the internals of git too much.

IPv6 experiments / lessons learned

During the last couple of days I did some experiments with IPv6 connectivity / applications / configuration.
For nearly two years I already got two sixxs.net tunnels. One for a server and one for my home connectivity.
I never got aiccu working on Mac OSX so the home tunnel was down most of the time.

Finally it got to me and I worked on getting 2 subnets now, again, one for the home network and one for the servers.
For the gentoo servers I used the router howto from http://www.gentoo.de/doc/de/ipv6.xml with the radvd configuration.
RADVD is a router advertisement daemon for ipv6 networks. IPv6 has a mechanism for auto configuration where the router advertisement daemon sends advertisements about the supported prefix (aka network/netmask in IPv4 world) and its own ip address for the gateway. So far it seems like most ipv6 stacks have this auto configuration included by default so every IPv6 enabled server in the reachable network suddenly has a IPv6 address. I never knew that that many servers of mine are IPv6-enabled and even quite some servers of my isp were suddently connected through IPv6 (getting me a curious call of my ISP ;-)).
Thats the first thing to be worried about, suddenly they are all connected to the big bad internet without correct reverse dns entries, firewalls and the like.
Speaking of firewalls, usually you don’t have a IPv6 firewall up at this moment. Your old ipv4 firewall rules won’t catch any ipv6 traffic. Therefore, again, every IPv6 enabled host is exposed to the world without proper protection. Thats even worse if you open a tunnel to your home network as the home network is most often connected through some router doing nat and internally just using private ip addresses so that the hosts are not exposed to the outside world at all. With opening the tunnel and enabling the radvd service you got them out in the open world either.

On my home network I got a CentOS5 server running which is doing some smb service and the like.
I got that one connected to the sixxs tunnel and started the radvd service on that box. So far so good, Mac OSX has IPv6 enabled with autoconfiguration by default so. So the hosts got the IPv6 addresses and routing.
ping6 worked (btw. nice to have most tools available as ipv6 cmds with just 6 at the end) but the browser delivered no IPv6 website. There you are, CentOS5 / RHEL HAVE a ip6tables ruleset enabled by default and that one was just open for icmp (ping) messages. Good protection but cost me a while to diagnose. So I opened some more loopholes for the IPv6 connection on the home network for smtp, imap, http, https and dns and still let the radvd daemon running.
At the server network I disabled the radvd service and manually set ipv6 addresses and gateway so that I won’t disturb neighbours in the network anymore :-). A strict ip6tables ruleset was enabled too.
For fun I went through the IPv6 certification by HE.net and got as far as to prove that I got:

  • ipv6 connectivity
  • an ipv6 enabled webserver
  • an ipv6 enabled mailaddress (yes my main mysnip.de mail address is now ipv6 enabled!)
  • reverse dns entries for my ipv6 enabled hosts (powerdns has no problems with that)

The step which still gives me trouble is that I can’t give fully ipv6 enabled nameservers to the outside world. My main nameserver is ipv6 enabled but the secondary ones from inwx.de don’t have ipv6 connectivity or AAAA entries so there’s not much I can about it.
Skimming through the maillogs on my mailserver I was stunned to see that *a lot* of spam is trying to deliver through IPv6 already. postgrey is working with ipv6 without trouble, amavis / spam assassin too so there’s not really a problem. Seems like spammers adapt more quickly to the new technologies though. On the other hand I found that freenet.de (a german ISP) got its mailservers connected through IPv6 already and is publishing AAAA entries for them. Therefore some mail is already delivered through IPv6.
In the near future I might try to offer some experimental IPv6 access to the services provided but without any native ipv6 connectivity (anyone knows if TeliaSonera is offering it and if it poses additional costs?) that doesn’t make too much sense for production.

At least now I can check how the applications I’m using and providing are working with IPv6. Also Phorum needs to be checked for that.

MySQL in Gentoo …

Merely as a note to myself I just looked around the Gentoo bugtracker to learn about the current state of MySQL in Gentoo.
So far I found two related bugs:
About 5.0.x
http://bugs.gentoo.org/show_bug.cgi?id=279493

What I learned from this bug:
Recent dev-db/mysql versions contain most of the percona patchsets already (neat!). Dunno about xtradb so far.
Latest in tree is 5.0.84 which I’m trying on a backup system now (even though is marked as ~x86/~amd64 aka testing.

About 5.1/5.4
http://bugs.gentoo.org/show_bug.cgi?id=194561
5.1 is going to be put into dev-db/mysql too (not in dev-db/mysql-community as it was before because of the changed development model)
Quoting Robin Johnson:

“I intend to issue a package move after we’ve had a few versions >=5.0.83 in stable, but there is no further need to make new -community ebuilds.”

So there will be 5.1.x in tree once some more recent 5.0.x versions were released as stable. Latest stable mysql version in tree is 5.0.70.

Seems like he’s also keeping back because of some more breakage in earlier 5.1 versions.
As another quote:

“I’m aware that 5.1.30 is out. However it’s still in bad bad shape. […] It certainly ate some of my data when I tested it.”

For the topic whats keeping them from stabilizing later mysql (> 5.0.70) versions I found some quote from Robin Johnson too (who seems to be THE mysql maintainer in Gentoo):

My most defining test for putting MySQL builds in the tree has been that it
passes both of the following:
1. Passes it’s own testcases (upstream has been atrociously bad at this, see
status2 in 5.0.72 for example)
2. It doesn’t eat my data or break my systems.

#1 is pretty easy as a start point, seeing if it works.
#2 is a lot tougher:
– 5.0.70 is the best option for now.
– 5.0.72 breaks most of the statistics code out there really badly (I filed
upstream bug 41131) – changes to SHOW behaviour as well as the ‘Questions’
variable.
[…]
Having upstream do sane releases is part of why there has been so long between
my 5.0.x bumps, because they haven’t passed my personal testing.

So far its mostly general stuff and I don’t know if MySQL-5.0.x still doesn’t pass its own testcases or anything. I couldn’t find more detailed stuff in the bugtracker.

MySQL proliferation

Long time no post but thats some stuff lingering in my head for a while looking at the mysql ecosystem lately.

For a long time I had to stay with MySQL-4.0 (sick I know) but there’s a lot of software to adapt to the new version(s) but lately I’m pushing it more and more into mysql 5.0.
MySQL 5.1 would be also interesting and even MySQL-5.4 … but thats where the trouble starts.
MySQL-5.4 came out of the dark, no one expected it and it gave numerous improvements … though its still in beta.
With that release (or at least around this time) MySQL started to change its release model to something new where the version numbers matter far less and where there should be regularly released. Well, Oracle has bought Sun which owns MySQL … so we will see what the next “release model” will be.

On the other hand there are some “forks” of mysql out there which offer more improvements or at least they are supposed to do this.
For one there is XtraDB which is supposed to be just a replacement for the InnoDB-plugin now … while it had more patches to the main mysql server before as far as I remember – so its not really a fork, just another storage engine.
Edit: just found a newer release – its called “MySQL with Percona patches”

Then there is MariaDB which should be a “… community developed branch of the MySQL database that uses the Maria engine by default.” (quote from the linked page) which is developed by Monty Program AB and the OpenDatabaseAlliance.

Being “only” a collection of patches and builds of MySQL with patches is OurDelta.
I’m pretty sure that there are more forks or patch collections out there, please forgive me if yours isn’t listed.

But here is the question. Which MySQL version / patchset / fork should one use?
Previously it was just the question of using the commercial enterprise or the community version. Now I’m just confused.

Next problem is the distribution on linux …
Nearly all my servers are using Gentoo as the linux distribution but have you ever looked at the status of mysql in there?
The latest “stable” version is dev-db/mysql-5.0.70. dev-db/mysql-community-5.1.21_beta is in there, but not marked as stable (5.1.39 being the last one on the mysql homepage), 5.4.x is nowhere to be seen, same for XtraDB, MariaDB … . I don’t blame it on the maintainers – who should keep up with that flow of releases, different branches and/or forks. I also heard rumours that all the latest versions are failing numerous tests for the maintainer(s) and therefore won’t go in.
If I want to use a recent stable version or one of the enhancements I will have to do my own builds instead of using the great gentoo infrastructure for that. I could, sure, but time is low and I would get far further if I could use the regular way to install (and update) software on Gentoo. I don’t have a full blown infrastructure team to keep the systems going.

If I wouldn’t have too many software using raw database calls in php or perl I would seriously consider switching to some other database, PostgreSQL or the like. I heard even Maurice considering switching to Postgresql in the future and supporting the Postgresql layer for Phorum.

If there is no one stopping this proliferation of the mysql ecosystem and providing reliable and regular stable builds of a tree, I think Oracle won’t have to worry about MySQL anymore. There will be not too much left of its userbase and community.
But maybe its just me, painting things black, who knows ;-).

The next steps for Phorum

Now that Phorum-5.2 has finally gone stable there will be hopefully some better modules as the possibilities have been vastly increased. One of the new modules for 5.2 which show quite some of the abilities is the rewritten user-avatar module for 5.2.
With modules you can use now (not everything is new in 5.2!!!):

  • use a supported API for files, users, custom profile fields and similar stuff
  • ability to hook the module-css into the css loaded by phorum for valid (x)html pages and not loading it separately (saving requests)
  • ability to hook the module-javascript into the javascript loaded by phorum for … see above 😉
    (both can use raw files, templates, functions for including it)
  • can do database calls without writing database dependent code (could still be because of the queries themselves)
  • use module-templates which are included in the module itself, no need to copy them to the template folder(s)
  • language files in the modules themselfes
  • adding controlcenter panels without copying files around

Also our module list for 5.2 is now auto-generated from the modules posted into the 5.2-modules forum in the right format.
Make sure to add categories too as listed in the docs!

So, now that we (could) have better modules, whats next?

Dan Langille has been working on a postgresql-layer for 5.2/5.3 which will probably be included in one of the later 5.2 release as a beta of this layer.
The next big release will be Phorum-5.3. Our plan for Phorum-5.3 is “just” to add even more APIs, changing large parts of the backend without touching much of the frontend code.
Therefore templates from 5.2 should work without a hitch with 5.3. Maybe there will be added features missing in the old template but otherwise it will continue to work as before.
I know we made it hard for some admins with the switch from 5.0 to 5.1 and 5.1 to 5.2 but all these changes were done for flexibility in the templates and making them far more consistent and therefore easier to implement.
Some of the APIs will be about forum handling and similar stuff so that you can build a new admin or an admin in another page far easier than before.
As usual you can see the tickets on the table for 5.3 in our ticket-list (from the 5.3-milestone).

And further in the future?
There is lots and lots of stuff in the ever growing Ideas-milestone.
We’ll see if any of this will see the light in 5.3 already or in a later release but we surely won’t get bored ;-).
I’m pretty sure that lots of stuff will be done at the MySQL Conference 2008 like last year where we’ve been coding and presenting there with lots of feature tickets closed for 5.2. You can help us to get there with donating to phorum.org!

laws and the use of logging IPs

in the light of recent court-decisions in germany ( german article ) which essentially disallows logging of IPs I’m wondering what one would really need it for?

I’m using IP-logging/-tracking in multiple ways:
1. statistics about visits and recurring users
2. storing it with forum-posts to allow law enforcement in case some user really goes over the line
3. tracking requests in a given time by IP to automatically block potential attacks

So what of that could be avoided?

For 1. , one could just ignore logging the ip but trying to count visits and recurring users would be impossible with that. What now? Maybe logging a md5/shaX of the ip to have some unique key per IP? Wouldn’t that still fall under the rule from the court as you could find out which was the actual IP?
Counting visits is an important tool for getting advertisers to advertise at a page (In my opinion). Any ideas?

For 2. , guess one could disable that but would I be responsible then for each and every forum-post because the real poster can’t be retrieved? (Yeah, laws in german are bad for the one offering the forum after all 😦 )
On the other hand there is the upcoming data retention ( german news collection about this topic ) which is planned for keeping all records for 6 months (!!!). So for now I should remove all tracking of ip-addresses just to be forced to store it for 6 months a while later?

For 3. , this behaviour gives me another problem too. Trying to load-balance over multiple webservers usually goes through a reverse proxy in front of the webservers which would always give the REMOTE_ADDR of the reverse-proxy to the apps. So the reverse-proxy would need to add this security layer. But I really failed to find one doing this up to know.
But is that really needed and I’m just oversensitive in this area? Do I need to accept any number of requests/s from any user?

Are there other use-cases for logging IPs?

How are other users handling this?

The editor of choice …

… yeah, everyone got his own idea of which editor he should or would use – thats the freedom of choice ;).
Brian loves his jedit, Maurice uses his VI (and can’t live wout vi-bindings and -code-folding) and I, I’m just going with the masses ;).
Currently I’m using Eclipse/PDT, coming right from the Zend IDE/Studio.
There were quite too many bugs in the current Zend Studio which I couldn’t live with (no, I don’t want to restart the IDE every half hour just because it forgets to show the content of the files) and PDT was just on its way to get to a final 1.0 so I used it.
Coming from Zend Studio its easy to use and for missing features in the IDE you can simply install some eclipse-extension – thats the power of using a generic IDE.
One thing I’m missing in PDT in relation to the Zend Studio is the line wrapping. There simply is NONE in Eclipse. Guess it was to teach coders to write 80cols code ;).
But for now I HAVE code which is far longer than 80 or even 160 cols and I don’t want to scroll around or reformat if I’m looking at a longish condition.

I also tried jEdit, Kommodo or the likes. I for one really want that project handling with function lists for the project, the possibility to just select a function and jump to its definition, having the comment of the function shown in a tooltip when using/typing it. Thats what I expect from an IDE.
Yeah, I know. These huge java apps can get slow sometimes but at least we got something to use our CPU’s for, eh? 😉

finally it has arrived: Phorum-5.2.0-alpha

It took quite some time but finally we made it and released phorum-5.2.0-alpha.
Brian already posted most of the new features in his blog .

One of the new things that don’t show up in the phorum-code itself are the revamped docs.
These are written in docbook-xml and are available (rebuild from trunk every hour) on phorum.org
as html-docs
and pdf: admin.pdf , developer.pdf , faq.pdf and the still empty user.pdf .
Therefore I renew my plea for help in this area.
It would be really great if you could help us to improve the docs. Every little thing helps.
Send us questions (and answers) for faq-items, texts for the user-manual and so on. Just send it in as plain text, we will convert it to docbook if you don’t want to mess with it.
Email-address for all docs-stuff is documentation@phorum.org (which will reply you at your first mail with a confirmation required).
If you want to play directly with the docs-source just checkout the trunk-tree as described in the wiki and look at docs/docbook in there.

Oh and before I forget: remember its alpha-quality. Don’t use it in production yet!

Thats one PHP-5.2.x feature for Phorum-5.2 I’d like to use …

Its the httpOnly Cookies support in the setcookie-call.
Now that Firefox 2.0.0.5 supports it too (as mentioned here and here)
the main browsers are supporting it.
Internet Explorer seems to have been the first one supporting it, with Firefox now and Opera meant to support it in 9.5.

Therefore it makes sense to use it now. Even browsers not supporting it are just ignoring the additional flag.
In PHP that flag was introduced in PHP-5.2 – another cause for going php-5.2 and up only ;).