Search

Linux Filesystems, Part 4 – Ext4 vs. Ext3 and why Delayed Allocation is Bad

It covers the main differences between ext3 and ext4 with a focus on filesystem consistency. This article was the initial motivation of this blog series, because many engineers are unaware that the standard option of ext4 (delalloc) is dangerous for their data!

Read more

Linux Filesystems, Part 1 – Block Device and Filesystem

It explains how files are split into blocks and how the filesystem acts as an abstraction layer between files (seen by applications) and blocks (physically written on disks). It also says why disks are called block devices.

1.1 Splitting Up Files Into Blocks

A block device is a storage media, like a hard disk. You can use ‘blkid’ and ‘fdisk -l’ to show a list of block devices attached to your system.

In the following example, ‘sda’ is a disk with 2 partitions ‘sda1′ and ‘sda2′, and ‘sr0′ is a compact disc.
A filesystem can be created on a partition and also on a whole partitionless disk, as both partition and disk are valid block devices.

blkid
  /dev/sda1: UUID="5b0e4487-b5aa-4186-81fd-f561308f4cf1" TYPE="swap"
  /dev/sda2: UUID="ec851750-9854-4227-8898-d5695dbd297e" TYPE="ext3"
  /dev/sr0: LABEL="VBOXADDITIONS_4.2.16_86992" TYPE="iso9660"

fdisk -l
  Disk /dev/sda: 12.9 GB, 12884901888 bytes
  255 heads, 63 sectors/track, 1566 cylinders, total 25165824 sectors
  Units = sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  Disk identifier: 0x000be060

  Device Boot Start End Blocks Id System
  /dev/sda1 2048 786431 392192 82 Linux swap / Solaris
  /dev/sda2 * 786432 25165823 12189696 83 Linux

 
In order to use the hardware efficiently and to increase the addressing limit the disk read and write operations are not processing the data byte by byte. Instead these operations are using a larger unit called a block (also called a sector), which is usually 512 Bytes.

Read more

Howto: Linux Filesystems

This blog series aims to be a reference guideline by collecting useful knowledge and tutorials about Linux filesystems, mainly about ext3 and ext4.

Because, as systems engineers/administrators, when we need to manually perform disk maintenance on a server, we know that a tiny mistake may have large consequences.

“Trial-and-Error” is sometimes a good way to learn, and sometimes not. For sensitive topics like internet security or filesystem operations, it is often better to understand the theory based on reliable sources before trying; in this way we can avoid putting the data integrity at unnecessary risk.

Read more

Howto – local and remote snapshot backup using rsync with hard links

Introduction

Your Linux servers are running smoothly? Fine.
Now if something incidentally gets wrong you need also to prepare an emergency plan.
And even when everything is going fine, a backup that is directly stored on the same server can still be useful:

  • for example when you need to see what was changed on a specific file,
  • or if you want to find the list of files that were installed or modified after the installation of an application

This article is then intended to show you how you can set up an open-source solution, using the magic of the famous ‘rsync’ tool and some shell-scripting, to deploy a backup system without the need of investing into expensive proprietary software.
Another advantage of a shell-script is that you can easily adapt it according to your specific needs, for example for you DRP architecture.

The proposed shell-script is derivate from the great work of Mikes Handy’s rotating-filesystem-snapshot utility (cf. http://www.mikerubel.org/computers/rsync_snapshots).
It creates backups of your full filesystem (snapshots) with the combined advantages of full and incremental backups:

  • It uses as little disk space as an incremental backup because all unchanged files are hard linked with existing files from previous backups; only the modified files require new inodes.
  • Because of the transparency of hard links, all backups are directly available and always online for read access with your usual programs; there is no need to extract the files from a full archive and also no complicate replay of incremental archives is necessary.

Read more