:::: MENU ::::

Posts Categorized / Linux

  • Mar 25 / 2014
  • 0
Linux

Download Flash Video Fragmented (F4F/F4M)

Today, for network usage optimization purposes, a lot of videos are streamed and sent to the users as fragmented packets instead of one file. According to Adobe, F4F file is Flash MP4 Video Fragment created and used by Adobe Media Server for HTTP Dynamic streaming like CloudFront, Akamai, … If you want to download this video, you will have to download the script file KSV (download this script on Github) and execute it on a system with php enabled (install PHP on Windows).

Once you’re on the page where video is loading, you have to look for manifest file in source code or network exchanges. This file has F4M extension, correspondig to manifest, which is the file containing the inventory of all the video fragments. It will be necessary for the script execution.

Here is an example of how the command-line should look like (you can modify the quality parameter, and you have to specify the path for manifest file, and output file):

php ksv.php --quality high --delete --manifest 'LINK_TO_MANIFEST.f4m' --outfile 'OUTPUT_FILE.flv'

Here are the different options you can use to customize command-line:

 --help              displays this help
 --debug             show debug output
 --delete            delete fragments after processing
 --fproxy            force proxy for downloading of fragments
 --play              dump stream to stdout for piping to media player
 --rename            rename fragments sequentially before processing
 --update            update the script to current git version
 --auth      [param] authentication string for fragment requests
 --duration  [param] stop recording after specified number of seconds
 --filesize  [param] split output file in chunks of specified size (MB)
 --fragments [param] base filename for fragments
 --fixwindow [param] timestamp gap between frames to consider as timeshift
 --manifest  [param] manifest file for downloading of fragments
 --maxspeed  [param] maximum bandwidth consumption (KB) for fragment downloading
 --outdir    [param] destination folder for output file
 --outfile   [param] filename to use for output file
 --parallel  [param] number of fragments to download simultaneously
 --proxy     [param] proxy for downloading of manifest
 --quality   [param] selected quality level (low|medium|high) or exact bitrate
 --referrer  [param] Referer to use for emulation of browser requests
 --start     [param] start from specified fragment
 --useragent [param] User-Agent to use for emulation of browser requests

All the original sources are available on GitHub : https://github.com/K-S-V/

  • Mar 18 / 2014
  • 0
Linux

Install VMware Tools on ESXi for Ubuntu Guest

If you want to install VMware Tools on ESXi for an Ubuntu Guest, there is two possibilities:

  1. Use the package provided on distribution (but can be not up-to-date and so not approved)
  2. Install directly from official sources provided in vSphere Client (will be automatically recognized)

Use the pre-package distribution

This solution is the easiest way to install tools. You just have to use apt on your guest to install it:

sudo apt-get install open-vm-tools

Use the official sources available in vSphere Client

This solution is not as simple as the previous one but is really sure and well-guided.

  1. Connect to your ESXi server using vSphere Client
  2. At this moment, ensure that you have a CDROM hardware available for your VM Guest (in Settings)
  3. Start your VM Guest running Ubuntu
  4. Check that you have already needed packages installed (most of time, they are not installed with a standard configuration):
    • gcc
    • build-essential
    • binutils

    You can use this command line to install them:

    sudo apt-get install gcc build-essential binutils
  5. You can now connect VMware Tools on your VM Guest using vSphere Client:
    • Right-click on your VM name
    • Select “Guest” > “Install/Upgrade VMware Tools”
    • Wait a few seconds

    This action will automatically (and transparently) mount the VMwre Tools CD into your VM Guest Ubuntu.

  6. Open a console on your VM Guest to mount the CDROM you just connected:
    sudo mount /dev/cdrom /mnt
  7. Copy the VMwareTools-x.x.x-xxxxxxx.tar.gz file from the CDROM to a local folder (‘x’ values depend on version used):
    cp /mnt/VMwareTools-9.0.5-1065307.tar.gz /tmp/
  8. Unzip the VMware tools package you just copied:
    cd /tmp/
    tar xzvf VMwareTools-9.0.5-1065307.tar.gz
  9. Go to the new folder created while unzipping file and run the installer script (perl script):
    cd /tmp/vmware-tools-distrib
    sudo ./vmware-install.pl
  10. From now, just follow the instructions displayed on the screen to install VMware tools.

Once all of this is done, you will see on vSphere, in VM details, that the vMware Tools are correctly installed.

  • Mar 07 / 2014
  • 0
Linux

Handling HTTP errors with Apache and Tomcat using mod_jk

When you are working with web server in frontend (Apache in this example) and backend application servers (Tomcat here), you may wish to customize your error pages depending on the HTTP error code returned (either from the web server or application server).

Pre-requisites

For the following example, you will need as pre-requisites at least those versions:

  • Web Server Apache >= 2.2.x
  • Mod JK >= 1.2.27 (rule extensions are not available in earlier versions)

Apache configuration

If you want to handle any error coming from your web server or from your application servers, you will need to perform some changes in your configuration to allow the handling. First of all you will need to set up some parameters in your Apache configuration file (in apache2.conf file directly or in your VirtualHost definitions).

In my example, I am handling any HTTP error code between:

  • HTTP 400 and HTTP 405
  • HTTP 500 and 504

I am using an only file (written in PHP) allowing me to send back a simple HTML page to the user depending on the HTTP error code received.

Here is what I’ve configured in my apache2.conf file for handling any error listed above:

ErrorDocument 400 /errorhandler.php?error=400
ErrorDocument 401 /errorhandler.php?error=401
ErrorDocument 402 /errorhandler.php?error=402
ErrorDocument 403 /errorhandler.php?error=403
ErrorDocument 404 /errorhandler.php?error=404
ErrorDocument 405 /errorhandler.php?error=405
ErrorDocument 500 /errorhandler.php?error=500
ErrorDocument 501 /errorhandler.php?error=501
ErrorDocument 502 /errorhandler.php?error=502
ErrorDocument 503 /errorhandler.php?error=503
ErrorDocument 504 /errorhandler.php?error=504

The errorhandler.php file could be like that (feel free to customize it as you want with a beautiful CSS theme, for the example I’ve used a basic configuration ;)):

The HTTP request could not be understood by the server due to malformed syntax.
The web browser may be too recent, or the HTTP server may be too old.";
        }
        if($_GET['error']=='401'){
                $errorcode = "Error 401";
                $errormessage = "The request requires user authentication.
This means that all or a part of the requested server is protected by a password that should be given to the server to allow access to its contents.";
        }
        if($_GET['error']=='402'){
                $errorcode = "Error 402";
                $errormessage = "Payment Required
";
        }
        if($_GET['error']=='403'){
                $errorcode = "Error 403";
                $errormessage = "The HTTP server understood the request, but is refusing to fulfill it.
This status code is commonly used when the server does not wish to reveal exactly why the request has been refused, or when no other response is applicable (for example the server is an Intranet and only the LAN machines are authorized to connect).";
        }
        if($_GET['error']=='404'){
                $errorcode = "Error 404";
                $errormessage = "The server has not found anything matching the requested address (URI) ( not found ).
This means the URL you have typed or cliked on is wrong or obsolete and does not match any document existing on the server (you may try to gradualy remove the URL components from the right to the left to eventualy retrieve an existing path).";
        }
        if($_GET['error']=='405'){
                $errorcode = "Error 405";
                $errormessage = "This code is given with the Allow header and indicates that the method used by the client is not supported for this URI.
";
        }
        if($_GET['error']=='500'){
                $errorcode = "Error 500";
                $errormessage = "The HTTP server encountered an unexpected condition which prevented it from fulfilling the request.
For example this error can be caused by a serveur misconfiguration, or a resource exhausted or denied to the server on the host machine.";
        }
        if($_GET['error']=='501'){
                $errorcode = "Error 501";
                $errormessage = "The HTTP server does not support the functionality required to fulfill the request.
This is the appropriate response when the server does not recognize the request method and is not capable of supporting it for any resource (either the web browser is too recent, or the HTTP server is too old).";
        }
        if($_GET['error']=='502'){
                $errorcode = "Error 502";
                $errormessage = "The gateway server returned an invalid response.
The HTTP server, while acting as a gateway or proxy, received an invalid response from the upstream server it accessed in attempting to fulfill the request.";
        }
        if($_GET['error']=='503'){
                $errorcode = "Error 503";
                $errormessage = "The HTTP server is currently unable to handle the request due to a temporary overloading or maintenance of the server.
The implication is that this is a temporary condition which will be alleviated after some delay.";
        }
        if($_GET['error']=='504'){
                $errorcode = "Error 504";
                $errormessage = "This response is like 408 (Request Time-out) except that a gateway or proxy has timed out.
";
        }
}
else {
        $errorcode = "Error";
        $errormessage = "An error has occured while trying to contact the website. Please contact support.";
}
?>


mod_jk configuration

Once you’re done with the main configuration of Apache, you will have to specify to Apache that you want to handle any HTTP error coming from the application servers. For that, you will have to use the rule extension use_server_errors when mounting an access point to a worker through mod_jk.

By default, for a mounting point you are using:

JkMount /myapp/* worker1

But if you want to handle any HTTP 40x error coming from this worker, you will have to add the use_server_errors extension to your mounting point:

# Use web server error page for any error
JkMount /myapp/* worker1;use_server_errors=400

To handle any HTTP 50x error:

# Use web server error page only for technical errors
JkMount /myapp/* worker1;use_server_errors=500

Or another option to handle both errors using just one command:

JkMount /myapp/* worker1;use_server_errors=400,500

You are now handling any error received from your application server with your web server allowing you to customize it as you want.

Feel free to read more about this rule extension on the official website.

  • Mar 06 / 2014
  • 0
Linux

How to compile and install its own Linux kernel

The Linux Kernel is the operating system kernel. It is the core of the system and provide an interface between the hardware and the software layers.

Most of Linux distributions are coming with a pre-compiled and ready-to-use kernel for most of standard usage. However, some functionalities are sometimes missing in the kernel to allow the execution of specific functions or the support of some hardware (cards, …). In these cases, you will need to use the source code of the needed kernel, recompile it and reinstall it to replace the existing one. All the official soruces of the different kernels are available for free on the website kernel.org.

I will show you here how to modify the source code, compile and install your own kernel for your system (command-lines provided are working under Debian/Ubuntu based system, you will probably need to adapt them depending on the system you are running).

Pre-requisites

In a first time, check that you got the different packages necessary for the operation:

sudo apt-get install build-essential make libncurses5-dev lzma initramfs-tools bc

Once these packages are correctly installed, you can go on the official website to retrieve the source and the configuration files for the kernel you want to use: http://www.kernel.org. In our example, we will use the kernel on version 3.13.5 (latest stable version available today).

Let’s move to the /usr/src directory and then download the kernel archive that we will uncompress immediately:

cd /usr/src/
sudo wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.13.5.tar.gz
sudo tar xzvf linux-3.13.5.tar.gz

Kernel sources are now uncompressed in the /usr/src/linux-3.13.5 folder and we can go ahead with its configuration.

Configuration

Let’s move in the directory you just created and start with the use of command make menuconfig to launch the configuration tool:

cd linux-3.13.5
sudo make menuconfig

At this time, you can activate any feature you want on the kernel. For example:

  • To activate the virtualization and its specific functions, move on Virtualization and then press the Space bar to get access to the different available functionalitiesuis  (that you can activate/deactivate as you want)
  • To activate the modules support, move on Enable loadable module support and choose the modules you want to enable by pressing the Space bar (activated items will be checked)

kernel_01

Once you’re done with this configuration, choose Exit and confirm the save of your configuration. This one will be saved in the current directory under the .config file (that you can manually edit with a simple text editor if necessary).

Compilation

Now that the kernel is configured, you will have to compile it so that it can be used on your system. For that, you will need to execute several commands for the different compilation steps:

  • Dependencies compilation (now useless with latest versions)
    sudo make dep
  • Kernel image compilation
    sudo make bzImage
  • Modules compilation
    sudo make modules
  • Modules installation
    sudo make modules_install

At this time, your kernel has been compiled and is ready to be set up on your system.

Installation

To install your own kernel, you just need to execute the command-line below:

sudo make install

This command will create the following files in the /boot start directory:

  • vmlinuz-3.13.5 : the current kernel
  • config-3.13.5 : the kernel configuration file
  • System.map-3.13.5 : the symbol table exported by the kernel
  • initrd.img-3.13.5 : the root file used temporarily during the boot process

This command should update automatically the Grub configuration file (grub.cfg). You won’t need to perform the update manually.

If it’s not done automatically, you can execute it manually with the command-line below:

sudo update-grub2

Checking

In order to check that your kernel has correctly been loaded and is used, you need to reboot your system. Once restarted, you can check which kernel is used with the uname command:

sudo reboot
uname -r
> 3.13.5

You are now using your own kernel with the functionalities you wanted.

  • Feb 19 / 2014
  • 0
Linux

Create Software RAID under Linux system

A hard drive disk is a device with limits (performance, lifetime, …) and a broken disk often leads to a data loss, and sometimes with a loss of data more or less important.
In order to avoid that kind of inconvenience, there are several hardware solutions allowing disk replication, but most of time with a too high cost for a standard use (non professional). Fortunately, there is also a software RAID, which can be set up easily and quickly without any additional hardware (assuming you already own at least 2 hard drive disks) given that it’s working on a software layer between the hardware abstraction layer and the file system.

There are different levels of RAID ; the most common levels are:

  • RAID 0 : the “striping” allows to improve performance by splitting the IO requests on several devices in parallel (2 to n disks)
  • RAID 1 : the “mirroring” allows to write same data on several disks at the same time (to disks)
  • RAID 5 : the “striping and mirroring” allows the combination of RAID 0 and RAID 1 (3 to n disks)

I will present here how to set up a RAID 1 solution (mirroring) but it will be easy to adapt the following steps if you need another RAID level.

RAID 1 setup

For the setting up, ensure you have the mdadm package installed and ready on your system. For Debian/Ubuntu, you can use the following command:

sudo apt-get install mdadm

Prepare both disks in Software Raid (type 0xfd) thanks to the fdisk tool:

sudo fdisk /dev/sdb
sudo fdisk /dev/sdc

You can now shrink and erase both disks with zeros thanks to the command below:

mdadm --zero-superblock /dev/sdb /dev/sdc

Create your RAID 1 (or any level, by changing the level attribute):

mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1

Format this new partition in ext4:

mkfs.ext4 /dev/md0

Mount your partition you just created:

mkdir /data
mount /dev/md0 /data

To get an automatic mounting point after any restart of the system, edit the file /etc/fstab  and add this following line:

/dev/md0    /data    ext4    noatime,rw    0    0

You can now check and follow your RAID status, by using following commands:

cat /proc/mdstat
mdadm --query /dev/md0

RAID Monitoring

To ensure the RAID is always working correctly and be informed when a disk is out of order, it is possible to set a monitoring solution thanks to the common tools of mdadm with a mail alert system.

Check that the monitoring settings are correctly defined in the file /etc/mdadm/mdadm.conf with the sender address (MAILFROM) and the receiver address (MAILADDR):

# instruct the monitoring daemon where to send mail alerts
MAILADDR [email protected]
MAILFROM [email protected]

You will need a mail server enabled on your server (postfix, ssmtp, …) and listening on port 25.

To check and test this configuration, ou can easily perform a test by sending a mail report thanks to this command:

sudo mdadm --monitor --scan --test --oneshot

To ensure that monitoring is enabled for your RAID devices, just check that the –monitor option is correctly set by performing the following command:

ps aux | grep mdadm
>>/sbin/mdadm --monitor --pid-file /var/run/mdadm/monitor.pid --daemonise --scan --syslog

If it’s not, you will have to add the option in the /etc/default/mdadm file (DAEMON_OPTIONS).

  • Feb 12 / 2014
  • 0
Linux

Add a Swap File

When you’re running on a Linux System (or any Unix Operating System), it’s common to use a swap to complete and optimize the memory usage on the computer (especially for older ones). There are different ways to do that, you can:

  • Create a swap on a full partition system (most of Linux distribution will allow you to create it when installing the system or later using fdisk commands to create partition and activate it by following the same steps than below from #2)
  • Create a swap file later on any partition and activate it directly using command-line

I will present here the second case: the creation of a swap file (you will need to get root rights to perform most of the following commands).

  1. Create a file with dd command to create a 1024MB empty file (which will be used by swap).
    dd if=/dev/zero of=/swap1 bs=1024 count=1048576

    We are here writing a file of 1024*1024MB=1048576 blocks size with a Read/Write of 1024 bytes at a time.

  2. Now format the file to swap format
    mkswap /swap1
  3. Ensure that the rights access are correct on this file (for security purposes)
    chown root:root /swap1
    chmod 0600 /swap1
  4. Now activate swap on your file
    swapon /swap1
  5. To deactivate swap on your file, juste use the following command
    swapoff /swap1

You can now check your swap availability by using the top or free -m command.

Right now, you probably want this swap file to be mounted and activated automatically at the system startup to avoid to perform these steps manually each time ? No problem, juste edit the /etc/fstab file and add this following line:

/swap1 swap swap defaults 0 0

Your system has now a swap file ready to use !

Question ? Contact