A few days ago, Matt Lee of the Free Software Foundation had asked in the #theora IRC channel for help in making a DVD of the Patent Absurdity documentary.  Several people volunteered to help, and I was one of them.  I discovered later that this was related to the “Who should see Patent Absurdity?” campaign of the End Software Patents project (an initiative started by Brad Feld, I just discovered!).

Incidentally, we all ran into different kinds of problems, doing what should have been a rather simple task.  The sources of the problems ranged from simple bugs to rather weird video formats.  In this post I will discussed the various issues that I personally encountered and how, alhamdulillaah,  was able to overcome them.

The most straightforward way to make the video was to take an already decent-quality clip, the 1080p-resolution Ogg Theora clip from the Patent Absurdity website, and just run that through ffmpeg with the ‘-target ntsc-dvd’ option to get a DVD-compatible MPEG-2 video stream.  While this did actually produce a compliant video stream that I could burn to a DVD, the resulting video actually had some audio synchronization issues as well as some other problems related to ffmpeg’s handling of Theora’s drop frames (segments of video that are basically empty frames because the data does not change at all).  So, the resulting video was unacceptable.

My next attempt was to actually go from the original source material that Matt had provided some folks over at Xiph.  What’s interesting is that the format was a very odd, 1280×1080 resolution, 4:2:2 chroma-sampled DV-encoded video with a display aspect ratio of 16×9 – that was 25GB in size..  The only application that actually supports this format is ffmpeg, which means, also, it’s more widely supported than one might think.  It’s still an oddball format, though.  So, I ran it through ffmpeg with the same target settings as above.  However, ffmpeg’s ntsc-dvd target does not change the chroma sampling format of incoming video, so it produced MPEG-2 video that had 4:2:2 chroma.  I’m not sure if that’s even valid for MPEG-2, but it’s surely not valid for a DVD.  So, I manually specified the proper chroma sampling using ‘-pix_fmt yuv420p’ (this means planar 4:2:0 chroma sampling).

Sure enough, this produced a DVD-compliant MPEG-2 video stream, but with one problem.  The aspect ratio was not preserved.  This resulted in video that was squashed horizontally.  And, try as I might, nothing I could do with ffmpeg with the MPEG-2 video stream from this point on could correct the aspect ratio, and using a utility, such as mpgtx, to fix it, would make it incompatible with DeVeDe, the application I’m using to master the DVD Video itself.  So, I had to do an intermediate step.

I deduced that the problem was rooted in too many transforms being required at the same time.  Part of the problem may very well have been my own inexperience with ffmpeg as well, as perhaps someone may be willing to clarify via comments.  Regardless, I decided to tackle the problem by doing the steps separately – rescale the video & resample chroma as one step, and then do the MPEG-2 encoding via DeVeDe, which will know how best to make DVD video it finds agreeable.  So, starting with that mutant DV video source clip, I extracted the audio out into a FLAC file (the source was 48k/16b/s PCM, so very high quality).  Then I worked on the video – I scaled it down to 720×480 with a 4:2:0 chroma sampling but with an aspect ratio of 16×9, and I encoded it with the lossless FFV1 codec developed by the ffmpeg developers themselves.  This gave me a 7GB AVI file – not small, but far more manageable than the original 25GB source file.  I downloaded that locally, and then I attempted to remux the video & audio back into the same file.  And this is where the next issue occurred.

I first noticed the problem on my server, where I attempted this technique first – that is, remuxing the FFV1 AVI with the FLAC audio into another AVI file, this time with combined video & audio.  I noticed my server started to lag tremendously, and even my rsync download of the video to my PC at home had halted (at more than 90%, no less).  I managed to kill the ffmpeg process, and upon inspection of the load of my server, it had apparently jumped up to 30 or so during that time.  I was confused, but I didn’t piece it together yet.

After the FFV1 AVI file had finished downloading to my PC (I had to restart the rsync, but thankfully, I had [and usually always] used the –partial-dir parameter to save partial downloads for just such an occasion), I attempted the same remuxing operating on my PC.  Less than a minute later, I noticed that my PC had frozen.  I knew it was still working because the hard drive was thrashing, so I attempted to SSH in to my PC from my laptop, which I happened to have handily next to me.  After wait for about 1/2 an hour for the connection to establish and actually be able to get the system to respond, I discovered that ffmpeg’s memory usage had jumped up to 1.3 GB.  And, as my PC has only 2GB of RAM and I had numerous bloaty apps open already (Thunderbird, Firefox, Konversation, Pidgen, Gwibber, and, of course, Gnome itself), this mean the system had run out of memory and was thrashing the swap file.  I managed to killall -9 ffmpeg, and that brought the system back to the land of the living.  I inquired on #ffmpeg why this might have happened, and the response I got was that ffmpeg doesn’t like FLAC in AVI.  So I tried other container formats – Matroska & NUT, notable.  They both exhibited the same problem, but I was able to kill ffmpeg in a similar fashion much more swiftly.  I decided to just suck-it-up and converted the FLAC to WAV and ffmpeg muxed that without any difficult whatsoever.  My PC didn’t even break a sweat.

As a side note, following some additional advice from #ffmpeg, I used mkvmerge to mux the FFV1 AVI file with the FLAC audio file, and it worked, but it lost the aspect ratio.  I tried to manually specify the aspect ratio, and it gave very odd numbers that were, perhaps approximations of 16×9, but I wasn’t satisfied, so I stuck with FFV1 & raw PCM audio in AVI.

Finally, I can jump into DeVeDe and attempt to make the DVD.  As I am quite pedantic about quality (in case you haven’t gathered that by now), I chose the best quality options for encoding the DVD video that DeVeDe gave me – highest rates for audio & video and two-pass video encoding.  Matt had specifically said he didn’t want any menu for the DVD – he just wanted it to play, so I removed all menu options.  The whole mastering & encoding process took abut an hour and a half, according to what DeVeDe reported at the end, but I just slept through that (I had stayed-up all night, partially because of this, but not solely so).  I burnt the ISO that it generated onto a DVD-RW, tested it on different systems (Fedora & Windows), including a consumer DVD player, and, alhamdulillaah, it worked beautifully everywhere I played it.

So, there you have it – the trek to make a DVD about the problems caused by software patents completely with free software.  It was a fun experience, and I’m glad that I was able to help with getting this important message out.  In shaaʾ Allaah this will have a positive impact on patent reform and strengthen the arguments against the enforcement and even existence of software patents.

Assalaamu ʿalaykum wa Rahmatullaahi wa Barakaatuh

Alhamdulillaah, I’ve decided to get back into blogging. It’s been quite a long time since I’ve had much of a presence in the Blogoverse, and part of that is due to just not having the desire to go through, login, and making a post through the traditional WordPress administrative interface. However, until now, I still have not found a blogging client with which I’m totally satisfied. So, I’ve decided to go ahead and give lekhonee another shot.

One aspect for which I have not made an opinion yet is that the interface is basically plain text, and it allows you to post in raw HTML, which isn’t terrible. However, having a rich-text editor really isn’t that difficult to add, so, I’ll just have to see if it’s worth it or not. So, let’s see how it works-out.

Alḥamdulillāh, I’ve decided that, after upgrading WordPress, I should do a little housekeeping.  As usual, I find a list best presents the ideas I wish to express:

• I upgraded to WordPress 2.8.2 (mentioned here).
• I fixed some posts that had some typos (I usually am good about this, but these had slipped by).
• I deleted some links that really weren’t relevant to me anymore (i.e., Ruby on Rails) and I added some new ones.
• I changed to a new theme (comments welcome!).

I’d like to elaborate a bit more on the last point above, since it merits some discussion and the action is rooted in something I care about deeply.

Previously, for quite some time, I’ve been using the theme “Vidiyal“.  I’ve actually stuck with it for quite some time because of it’s simplicity, elegance, & beauty (in my eyes, at least).  However, a post by the author of the theme really made up my mind for me.  To put it shortly, despite the obligation to release derivative works of WordPress, such as a theme, under the GPL (as that is what WordPress is licensed under), he insists he does not have to.  I’m not so concerned with the legal issue, to be honest, but the insistence not to release the code as free software is what unsettled me.

To be honest, I didn’t really check before using the theme, and I guess I didn’t really care before this.  But now that the issue has been brought to my attention, I have to stick with what I believe is right.  So, I’ve stopped using all non-free (as in freedom) themes, and will only use themes that I know are free software.  I feel this is the better way to go.

I do hope that Sadish and others see that free software, both practically and, more importantly, fundamentally, are important concepts for protecting our freedoms as human beings.  Software freedom is something we should stand up for, and not tolerate being abused or disregarded.

So, with that, I close my rant and resume pretending to be productive.

Alḥamdulillāh, I’ve just successfully upgraded my blog to WordPress 2.8.2, along with several other blogs that I maintain.  I’m really looking forward to the merger of WordPress & WordPress-MU, which, ideally, will make managing multiple WordPress sites on the same server easier to deal with, in shāʾ Allāh.

Alhamdulillaah, I’ve discovered a new WordPress blogging client packaged with Fedora – namely, Lekhonee. There aren’t a lot of formatting options, but it appears you can post using HTML, so that’s a plus, I think. Let’s see how it goes.

Alhamdulillaah, WordPress 2.8 has been released!  Grab it from the downloads page or upgrade, like I do, via SVN.

Alhamdulillaah, the Fedora Project has released Fedora 11!  You can get the torrents and/or check out the new features!

(Sorry for the rather uninspired & lazy post…I usually like to share more details, but I’ve got work to do…;) )

Alhamdulillaah, I’ve successfully loaded the Fedora 11 Beta Live CD onto an SD card.  This allows me to boot directly from the card using my USB card reader, much as if it were a USB key/flash drive.  In short, after a few attempts, everything just works.

The few attempts mostly related to loading the Live CD image onto the SD card, as my flash memory reader is somewhat fussy and I was experimenting with different filesystems – my preference being to use a free software filesystem before using one of Microsoft’s.  I was able to do this using the ext2 filesystem (no need for journaling on a Live USB key, right?), as I have no need to store anything else on it.  Incidentally, I created an overlay image for persistence, so any changes made to the system while running should be saved and persist between reboots.  We’ll see how that works.

One of the driving factors for getting the Fedora 11 Beta Live image working is because I want to give my father, who just recently purchased a new netbook (Lenovo), a nice tour of a recent version of Fedora.  A Live USB version would be ideal for this.

Update: I’ve since gone ahead and installed Fedora 11 Beta on my laptop, as well (from the DVD install, though).

]]> http://www.basilgohar.com/blog/2009/04/06/fedora-11-beta-livecd/feed 0 http://www.basilgohar.com/blog/2009/03/13/hdr-scooter http://www.basilgohar.com/blog/2009/03/13/hdr-scooter#comments Fri, 13 Mar 2009 07:54:36 +0000 Basil http://www.basilgohar.com/blog/?p=442 Bismillaahir Rahmaanir Raheem

Alhamdulillaah, I am quite pleased with the following HDR image I was able to assemble.  While not as good as the best that are out there, it is definitely a step-up from my other attempt I shared here.

HDR Scooter

The workflow this time was almost the same as always, with one minor difference.  I took several shots of the same scene at different exposure levels (1/3 EV), amounting to a whopping 28 (!) separate exposures, which is about the most I’ve ever done.  I renamed the images so that they were in sequence from darkest to lightest, which helps with the next step, which is to align them using Hugin’s separately-packaged align_image_stack.  However, instead of using the align_image_stack utility directly, I called it from within Qtpfsgui (using the –align switch on the command line).  I used the default HDR generation settings and output to OpenEXR format.

The difference is that I did the HDR generation from within Qtpfsgui rather than using align_image_stack’s own routine (which also outputs in, I believe, Radiance HDR format).  Comparing the two, I realized that there wasn’t much difference at first, but at a closer zoom (e.g., 1:1), the Qtpfsgui method has far less noise in some areas (but not without some glitches, maybe you can find them), resulting in a smoother & more-natural picture.  I may consider playing with more than just the default HDR parameters with Qtpfsgui for generating the HDR image to see what other results I can get.

The final phase is tonemapping, which I generally don’t tweak.  Contrary to previous attempts, the Mantiuk algorithm produced a very colorful image, which I usually can only get with Fattal, which brings with it some other complications.  All in all, it was great.

If I had to guess, I would say that part of the reason for the ease of working with this HDR sequence was that the image had plenty of HDRness available to play with – the sky, the ground, the scooter with its reflective highlights as well as its darker spots.

Alhamdulillaah, to my pleasant surprise, I discovered that I started using Fedora personally with Fedora Core 4, and not 5 like I had previously thought.  This makes me feel better as I’m glad I started using Fedora earlier, because I like to think I recognized the importance of a completely (or as close as possible) free operating system (free here means freedom, not just free of cost).

At the time, I was content running it as a dual-boot system and just playing around with it.  By the time Fedora Core 5 came around, though, I was giving serious thought to using Fedora (Core, at the time) as my primary OS.  The complete transition happened, I believe, with Fedora 7 (which is also the year that Fedora Core & Fedora Extras combined, creating…Fedora!), where I relegated Windows XP to just being a spare OS for the very few things it does that I am unable to do with a completely free operating system (yet).

My experience with Fedora Core 4 was overwhelmingly positive.  I am sure I was impressed by the sheer amount of Free Software that was available for it as well as the elegance of a system that was not originated in Redmond.  I was and still am very pleased with the way Gnome works, and that was a big factor for me at the time as it is now.  I do wonder, though, if I downloaded & installed Fedora Core 4 today, what my experience would be link….hmm…

I remember seeing Fedora first in use as Fedora Core 2, when I was surprised someone was using it, as the name “Fedora Core” made me think it was the core of an operating system, but not the complete one.  I don’t know how or why I picked that up, but it is likely related to the perception that Fedora is not really meant to be used, and it is just a development platform (a blatantly false accusation).  I do know that I heard of Fedora Core when it came out, but again, my perception was likely incorrect at the time, thus causing me to adopt using it until quite late.  I had been using different GNU/Linux distributions since I was a freshman in college (can anyone say 1999?).  At the time, I recall trying “Corel Linux” – yes, I…don’t know what I was thinking.  I do remember it being based on Debian, though.  I also remember it significantly for a large amount of disk activity.  I don’t remember much else other than me deleting it shortly afterward.    Other flavors of GNU/Linux I tried were Red Hat, and that’s about all I can remember now.

So, anyway, this might be a good idea for a meme.  My first Fedora.  Talk about when you first started using it, why you did, and how the experience was.

I have the bad habit of starting things and not completing them.  This is bad for numerous reasons, just a few of which I’ll list shortly.  I will warn you now that as this post is meant to help me organize my thoughts, it will list-heavy, as I do find lists and other types of grouping structures to be beneficial and help in making relationships (e.g., database schemes).

I already know that as I type this I will revisit this post in the future as I will be unable, in one sitting, to be think of all of unfinished projects.  Therefore, I will come back and add to the list as I recall more things (and find the time and desire, a rare combination these days, to update the blog).

For

• Unfinished projects tend to clutter your mind
  • I tend to turn to unfinished projects whenever I work (i.e., get paid to do something) and hit a stumbling block.  This severely hinders my progress on work, which ultimately leads to less getting done.
  • Time when I could be brainstorming solutions is usually spent wondering about these myriad unfinished projects, which, lacking much focus, tend to yield few to no tangible solutions.  This includes walking/riding to the masjid, showering, and other activities that tend to leave the brain free and, ideally, well-suited for addressing beneficial thoughts.
  • If I try to focus on something, I can have a hard time preventing my mind from wandering to these unfinished projects.  This is related to the point about work above.
  • It’s just not healthy to have so many things on the mind at once.  There is a point at which the benefit of multitasking is overshadowed by the overhead of just trying to keep track of everything.  It’s not efficient.
• Having so many unfinished projects means that it can be hard to pick just one to work on.
• Playing catch-up may give a false sense of achievement, because the value of these unfinished projects are not equal, and therefore, I may complete one that is really not that important while one that is more valuable was never completed.  Therefore, just trying to clear out the backlog of projects may yield a net result of close to zero.
• There are many more, and I may amend this list in the future as I recall them.

So, here is the list of my unfinished projects (at least those that I can recall at one point or another).  I am including some thoughts on the different ideas as well.

• Redesign the Audio Islam website
• Write a script to automatically tag all the audio files, especially the ones containing Qurʾān recitation, with reciter & sūrah information
• The Fedora Documentation manpage coverage project
  • This is basically an idea I had related to the Fedora Documentation project to ensure that all executables had an associated man page (manual page, a low-level documentation system for Unix-like operating systems such as GNU/Linux).
  • The first step in this is to canvass the existing packages & executables and see what already has a man page.  I have already started on this, but there is much work left to be done.
  • There are plenty of sources for documentation that can be used as baseline manpages, including the Debian project.  So, this project doesn’t require as much work as it does just coordinating and facilitating.
• A Paltalk replacement
  • Paltalk is a voice conference & chat application (video is also supported, but I don’t care about that right now) that is widely used, but terribly proprietary.
  • My goal is to create or synthesize a replacement for Paltalk using only free software and open technologies
    • The best candidates for this are SIP (for multimedia) and Jabber/XMPP (for text chat).
    • The landscape for software that can do this in a simple fashion is rather bleak, but there are some solutions.
    • I’m currently focused on FreeSWITCH as the SIP server and ejabberd as the chat backend.
    • Linking them such that it will be one service will be quite challenging, and still requires a free software client that has solid support for both SIP & XMPP
      • I’ve been looking mostly at Qutecom, SIP Communicator, and Psi
    • • On GNU/Linux, the Empathy client for the Telepathy framework is outstanding, but, is limited to free-software operating systems.  I think Windows support is a long way out still.
• AlMaghrib in Malaysia
  • I am trying to catalyze the establishment of AlMaghrib Institute seminars in Malaysia (almost surely in Kuala Lumpur, at least at the beginning).
  • I’ve already purchased the domains almaghribinmalaysia.com/net/org, and I just need to get the site setup.
    • To start off, I think just a blog with some information about AlMaghrib is fine.  It can be fleshed-out later.
  • Alḥamdulillāh, there is already a lot of interest both within and without Malaysia.  I have several contacts here that are also excited and may be able to facilitate the first few steps.
    • I see the first step being getting a group of excited & enthusiastic locals to get behind the effort.
    • After that group is identified (and it is really an ongoing process), see about getting the support of existing organizations.
      • From an initial perspective, it seems like this is going to be easy, in shāʾ Allāh.
    • With this support, we will want to schedule some kind of event to gain a wider audience interest.  Something like a conference or teaser seminar, perhaps a week-long program with some of the AlMaghrib instructors.
      • Many of the AlMaghrib instructors are already known or well-known here
      • Having Suhaib Webb might be particularly nice, as his wife is Malaysian.
      • Waleed Basyouni has expressed an interest in coming to Malaysia, and some work is already in progress regarding that, but it may have stalled (temporarily only, in shāʾ Allāh)
      • Yasir Qadhi is another instructor that has some reknown here, apparently (from what I’ve been told).
• HidayahTech
  • My own technology consulting business
  • Although HidayahTech is currently the name of my blog, I have also resolved to use this as the name for my own consulting business.
  • Services will include
    • Audio recording, editing, mastering, and production (I did this, for example, for the Texas Dawah Convention in 2007)
      • I am consider splitting this and other audio-related services into it’s own business, namely, Hidayah Audio
    • Web technologies
      • Hosting (this is generally quite easy)
      • Moodle (an open-source learning mangement system, for putting schools online, for example)
      • Audio services, including hosting my “replacement for Paltalk” solution above
    • Consulting
      • This will include a wide variety of services ranging from system administration, general IT, infrastructure, to web development
      • I may take some partners or hire some friends for this aspect of this business, as I will need help for some points of it
• Global Homeschooling
  • This is my premiere project, if I had to pick one.  I consider this to be something great if I can achieve it, by the Permission of Allāh.
  • To sum it up, I want to develop a system whereby Muslims (and truly, anyone) can homeschool their own children easily.  Homeschooling may be too specific of a term, though, because I want the scale to include a plan that can be implemented by governments.  I would like children, and really anyone, to be able to be educated within their own homes, utilizing the bevy of technology that exists.
  • Technology creates options & opportunities, and this is a belief I take strongly.  Therefore, I think we should utilize it.  I see education as one of the most important investments that can be made, whether it is Islamic or related to the worldly sciences, both of which are important.  I actually believe the distinction to be artificial, but it will take time to get there.
  • So, to put it in a few words, I would like to develop a system of education leveraging technology that is globally accessible allowing education to be released from the confines of school buildings.
• Learn the Python programming language
  • I think learning Python has a high return value on the types of applications I would like to write, including graphical ones.  Also, it is quite the preferred language for many platforms, not the least of which is Red Hat & Fedora.
• Memorize the Qurʾān
  • Please keep in mind that this list isn’t in order of importance…;)
  • I think this is a goal every Muslim should have.  I started this goal, seriously, while I was staying in Egypt with my parents, even semi-regularly visiting the imān of the nearby masjid to review my memorization.  Sadly, since arriving in Malaysia I’ve not really resumed, and I am afraid I may start regressing.  This is the kind of project that can be achieved through a little, regular & constant work every day.  I really should commit to something, even if it is as simple as one āyāh per day.  That would be better than none.

Alhamdulillaah, I’ve just managed to produce my first HDR (high dynamic range) image the results of which I’m pleased with.  Once again, I have CHDK to thank for enabling this feature easily.

Note:  I begin this post with a mildly detailed explanation of the human visual system, HDR & LDR images, and the process to create HDR & tone-mapped images.  The pictures themselves are near the bottom of this post.

First, a little background about HDR.  The way our bodies capture and process imagery is, to put it lightly, amazing.  Starting from when light first hits the cornea through to when we perceive the visual object (not to mention how our brain’s store such visual images, which is far from just keeping a “bunch of bits”), the Words of Allāh resound with truth regarding His Creation:

And it is He who spread the earth and placed therein firmly set mountains and rivers; and from all of the fruits He made therein two mates; He causes the night to cover the day. Indeed in that are signs for a people who give thought. (Alqurʾān 13:3)

Thus, in the way that they are created, our eyes can capture an amazing amount of detail and range in their own way.  The range of regular (so-called “LDR” – low dynamic range) images is quite limited in comparison.  And this is, actually, the vast majority of image-related technologies, be they cameras, display devices, or even image formats such as JPEG.

HDR images, however, are those which do or have the ability to capture image details, specifically color & brightness information, at a much higher level than normal technologies.  Since normal technologies, such as our digital cameras and screens, can not capture nor display HDR images, we have to take a different route to get the information.  One technique is exposure bracketing where one takes a picture of the same scene at different exposure levels, one after another.  The reason for doing this is that, while the normal dynamic range of a camera may be unable to capture the bright, medium, and dark details of an image all at once, it does have the ability to capture these details separately with different settings.

Alhamdulillaah, there is also software that exists that allows you to take these separate LDR images and combine them into one now-HDR image that contains the details from all the separate exposures.  Doing so is not trivial, and it may take one multiple attempts to get something that looks good.  More often than not, the result is eery, because of a technique called tone-mapping.  Tone-mapping is a technique whereby the colors and range of an HDR image are brought down into a simulation of what our eyes would perceive in the scene.  In reality, that’s all but impossible, but it yields some interesting, beautiful, and sometimes striking results.

The specific software I use is the unpronounceable Qtpfsgui (I just say it one letter at a time).  Qtpfsgui is a front end for pfstools, which is an outstanding image toolkit for these kinds of images.  It takes care of aligning, merging, and finally tone-mapping the images one step at a time.  I short-circuited the first two steps (because Qtpfsgui can be fussy sometimes)  by using the command-line tool align_image_stack to align the different exposures (since I moved a bit between exposures) as well as creating the HDR image from the three separate images.  Then, I used Qtpfsgui just for the tone-mapping stage.

The first row of pictures are the original images, taken at -4, 0, & +4 EV settings.  Looking closely, you’ll notice that each range brings out different details: the darkest shows details in the clouds, the middle shows details in the sea, and the brighest shows details in the sand.  These are the separate LDR images at different exposures that I talked-about above.

The second “row”, containing only a single picture, is the result of the several steps I mentioned above to produce an HDR image.  I make no claims into how “real” this image looks, but it does show the details that I was looking for – from the clouds to the sand – in one image.  I also think it looks cool. 

This is the first HDR image I’ve created that I’ve wanted to share, and so I took this opportunity to explain the process too.  If you have any questions, then by all means, please post a comment and then subscribe to the comments so you’ll know when I’ve replied.

Note: This is for future reference as I try to make some progress on an old project.  I do not think the last value is a default for Fedora, but rather, something added later.  If someone knows otherwise, please correct me!

Update: I’m starting to dislike fixed-width templates…corrected the last $PATH value so it will fit on my blog without overflowing into CSS “Elsewhere”.

Update 2: Ahhh!  WordPress!  WYSIWYG, not WIWIWYG (What I Want is Why You Get)!

Note: This is a post I started writing back in November, but only got around to updating now.

Alhamdulillaah, I cannot believe I’ve lived for so long without knowing about screen (Wikipedia article).  To put it as simple as possible, think of screen as a “tabbed” interface for terminals/shells.  This is a weak analogy, however, because it enables so much more than tabs do for browsers, for example.  Onwards for more details.

History

I had first heard of screen through references and then through some of my friends, but I didn’t give it much heed because it seemed like something too complex for something so simple as a terminal application.  Little did I realize that, like most applications with their roots in Unix, it’s both simple and extremel useful.

Invocation

screen is invoked by typing it’s name.  It will appear initially as though you’ve reset your terminal, but this is just a new screen session, which is effectively a new login. Each screen window (which I will explain later) is independent and has it’s own scrolling buffer, independent also of your own terminal’s buffer. This means that screen as an application stores previously displayed text on the machine on which it is ran, so if you’re running it remotely, it won’t affect your local machine.

Basic Usage

Commands in screen are almost always prefixed by ctrl-a.  For example, creating a new window in screen is “ctrl-a c”.  Switching to the next window, if you have more than one, is “ctrl-a n”.  Previous is “ctrl-a p”.  screen windows can be exited by simply typing “exit” as if they were any other terminal session.  When the last screen window is exited, screen itself terminates.  Screen windows can also be killed by typing “ctrl-a K” (note the capitalization of “K”).

It’s also easy to jump to windows 0 – 9 by typing “ctrl-a” followed by the window number.  So, for example, if you want to jump to window 3 of your screen session, just type “ctrl-a 3″.

Detach and resume

The real beauty is using screen is its persistence.  Chances are, if you’ve used remote login shells frequently, you’ve been disconnected while working.  Most shells will respond by terminating whatever was actively running in the foreground of that terminal.  This means your vim session was interrupted.  It also means whatever service you had started is probably dead by now.  screen takes care of all of these things, because it behaves as a persistent application.  Thus, whenever you disconnect from the terminal, screen stays running, and it allows you to resume a previous screen session with the command line option “-R”, which means reattach an existing screen session, else start a new one.  Additionally, the “-d” argument might be useful, because it will detach the session from an existing login, in the case it is still attached elsewhere, such as a remote login from another location that you’re no longer using.

For example, if you are logged-in to your server from home with an active screen session, and then go to work, calling “screen -d -R” will detach the screen session from your home login and reattach it, where you left off, to your login from work.

Another advantage is that screen effectively allows you to make a daemon (background running process) from any command-line application, including those for which it normally isn’t possible.  An example of this would be rtorrent, which is a torrent downloading application normally meant to run in the foreground.  I personally like to start an rtorrent process running in one screen window and just leave it running.  Then whenever I login to that server and attach the screen session, I can check on the status of my torrents very easily by ctrl-a p/n to the appropriate window while doing other work.

Conclusion

So there you have it!  I realize this is a very introductory article with regards to screen, but it should be enough to get your started.  If you run into any problems with it, just type “ctrl-a ?” (that’s a question mark) and a friendly window will pop-up explaining, albeit briefly, all the different commands and what they do.

For a long time now I’ve been wanting to setup an LDAP back end for various Hidayah Online Network resources, especially the Columbus Dawah project.  This has become particularly pressing as the Columbus Dawah website has three primary public services – the blog, the forums, the school.  Additionally, in the future I want to roll out webmail (publicly) for volunteers & administrators within the project.  The problem with running these disparate services is that they all have their own authentication built-in, which means someone needs to register on each site independently to access any services that require one to be logged-in.  Therefore, a system that would unify user authentication for all services (i.e., LDAP) would save a lot of hassle for both the users and myself.

Native support

Naturally, the first challenge is preparing the different software packages that run the various parts of the Columbus Dawah website for utilizing LDAP for authentication purposes.  The relevant bits of software are WordPress for the blog, phpBB for the forums, & Moodle for the school.  Additionally, I will use RoundCube for the webmail, but since it will act as an LDAP client, it will not need any special setup on it’s own – the back end IMAP e-mail server (Dovecot) will be the one that interacts with LDAP.  That will be described in another post, in shaaʾ Allaah.

Of these mentioned bits of software, it appears that only WordPress comes without some kind of native LDAP support.  While this will be a challenge moving forward, one plug-in exists that at least offer some support, but I’m not sure if it’s updated for the latest version(s) of WordPress.

Structure

Although I haven’t finalized it, there is definitely the hope that I can use LDAP for authorization as well as authentication.  While authentication would allow me to let someone login to the site, authorization would allow me to give different users different levels of access.

For example, let’s say we have a simple organization that has three “ranks” of members – member, volunteer, & officer (this is roughly based on the plan I am working on for Columbus Dawah).  Furthermore, we want to grant different levels of access to the site’s resources depending on a user’s rank.  For a regular member, we would like to allow to him to login, post to the forums, post comments on the blog, and enroll in courses in the school.  For a volunteer, we would like to add to that the ability to have an e-mail account.  Finally, for officers, we would like them to have access to make blog posts and create & edit courses in the school.

Authentication alone would only allow us to grant or disallow access site-wide using one account.  Authorization gives us that additional feature of granularity to allow access to some or all of the site based on permissions.  LDAP supports all of this, but it is up to the underlying software to enable such features through LDAP authorization in a way that makes sense for the application.  As all the aforementioned applications have a concept of groups or roles with different sets of permissions or capabilities, and as I intend to utilize such features as the project grows, the ability to authorize users as well as authenticate them is tantalizing at the very least, and down right essential the more I think of it.

Fedora Directory Server

The specific LDAP server I intend on using is the Fedora Directory Server (website).  First and foremost, naturally, it’s open source.  And, the fact that I’m choosing an LDAP server that is related to the Fedora Project really should come as no surprise to anyone with even the slightest familiarity with this blog.

However, FDS is also quite an advanced LDAP server, supporting a slew of enterprise-level features while, according to most accounts, being reasonably easy to work with.  Therefore, it is quite a good choice, and may very well be the best one.  Time will tell how close to the truth that statement is.

What’s next?

So, what’s next?  Well, the most glaring obstacle right now is that I have zero experience working with and/or configuring an LDAP server, so I’m going to start by just playing around with the protocol and see if LDAP will work as smoothly as I hope it will.  If I feel comfortable enough with that, then my first foray into LDAP will likely be to configure my new e-mail server to use it as a backend.  That should prove to be quite interesting, moving forward.  So, stay tuned and I’ll do my best to keep things updated!

I have my new server (aalimraan.hidayahonline.net – the one hosting Audio Islam) setup to log web accesses each day to its own file which is then bzip2-compressed.  This is convenient for a variety of reasons.

• Firstly, I can easily access statistics for each day by processing the appropriate file.
• B, I can see the relative activity on each day at a glance by seeing the size of the file, keeping in mind it’s only a rough estimate since compression can skew the results (e.g., many requests for the same, exact URL may compress much more than fewer, disparate URL requests, resulting in a smaller file and a seemingly less active day)
• 3, by being compressed text files, they take up very, very little space.

The fact that the files are compressed using bzip2 means they are relatively tiny (by an order of magnitude compared with their uncompressed forms).  On top of this, I can easily access their contents using bzcat, which simply decompresses the files on the fly, allowing me to redirect the uncompressed text stream to whatever utility I’m hoping to process them with – such as grep, wc, or whatever.

So, I wanted to download the files in bulk to my laptop at home so I could write a PHP script that will process the access logs and store them in a database so I can extract all kinds of goodies from them such as which sites link to Audio Islam the most, which URLs are the most popular, and so on.  So, rather than download all the separate files individually, I wanted to concatenate the log files created thus far into one larger file so that I could just download it in one go.  Additionally, several text files concatenated together would naturally yield a smaller file (at least, so one would assume).

So, I ran the following command (or something like it):
cat access_log*.bz2 | bzip2 -c | ~/aalimraan.hidayahonline.org-access_log-20081226.bz2
The intention being that I wanted to create one large bzip2-compressed file as the concatenation of all the other, smaller daily files by first decompressing them into a continuous text stream (as if it were one large log file) and then recompressing them into the larger, single file.  But when I got around to processing that file on my local machine, the uncompressed output was garbage!  I was surprised, and then an idea hit me – I decided to run the uncompressed output through bzcat once again (the idea being that I am now decompressing it twice).  Lo and behold, the output of that invocation was something that looked astonishingly like a web server access log!  So what happened?

Look closely at my command, and you’ll see that I made the quite silly mistake of using plain old

cat

instead of

bzcat

.  Thus, I was recompressing a stream of already bzip2-compressed files, which is, to say the least, quite pointless.  In fact, the resulting file is all but useless to me, because I cannot really tell where one file begins or ends, so I only get output from the first bzip2 file.

Needless to say, let this be a lesson to make sure, when piping data around, you know what you’re doing.

This has been a public service announcement from you local system admin.  Thank you for listening!

Alhamdulillaah, while waiting for an answer an #fedora for the question of where the custom spellcheck files for hunspell, the unified spellchecking engine for Fedora 9 and upwards, reside, I discovered that they are stored in .config/enchant, which isn’t very intuitive, but makes sense after discovering that enchant is related to the Fedora hunspell integration.  Click to see the link that lead to this discovery.

Alhamdulillaah, I’ve (finally!) installed Fedora 10 on my laptop, but it’s not been without its bumps.  I cannot blame this on Fedora 10 nor the anything else related to the release.  But, in the end, I don’t have a DVD to install from, only the LiveCD (which I booted from my SD card reader), and that left me without a lot of packages and also without Ext4!  And now, after having copied my Evolution profile directory over, I discovered that the new version does (2.24.1) does not supported the “unmatched” search folder feature which I had come to rely on in recent weeks.  This is a real bummer as it all but nullifies the advantage of using search folders (for me, at least) in Evolution.  Alhamdulillaah, at least there’s a bug report open with one developer sympathizing.  However, a fix for it may not be forthcoming until the next release.

I think the time has come to take a look at Thunderbird again…

Alhamdulillaah, WordPress 2.7 has been released!  Hit the link for all the releasy goodness.

I’ve upgraded my personal blog, and I’ll take care of the rest in due time, in shaaʾ Allaah.  Upgrading with SVN is always recommended.

Happy blogging!