KernelNewbies:

Linux 3.4 has been [https://lkml.org/lkml/2012/5/20/126 released] on 20 May, 2012.

Summary: This release includes several Btrfs updates: support of metadata blocks bigger than 4KB, much improved metadata performance, better error handling and better recovery tools; there is also a new X32 ABI which allows to run programs in 64-bit mode with 32-bit pointers; several updates to the GPU drivers: early modesetting of Nvidia GeForce 600 'Kepler', support of AMD Radeon 7xxx and AMD Trinity APU series, and support of Intel Medfield graphics; there is also support of x86 CPU driver autoprobing, a device-mapper target that stores cryptographic hashes of blocks to check for intrusions, another target to use external read-only devices as origin source of a thin provisioned LVM volume, several perf improvements such as GTK2 report GUI and a new 'Yama' security module. There are also many small features and new drivers and fixes are also available.

TableOfContents()

1. Prominent features in Linux 3.4

1.1. Btrfs updates

This release has many Btrfs updates. Recommended video from Chris Mason, "[http://linuxfoundation.ubicast.tv/videos/btrfs-filesystem-status-and-new-features/ Btrfs status and new features]" (video file in [http://d17mmld7179ppq.cloudfront.net/btrfs-filesystem-status-and-new-features_7007df/hd_ready.webm webm], and [http://d17mmld7179ppq.cloudfront.net/btrfs-filesystem-status-and-new-features_7007df/hd_ready.mp4 h.264] format)

1.1.1. Btrfs: Repair and data recovery tools

A new data recovery tool ([https://btrfs.wiki.kernel.org/index.php/Restore btrfs-restore]) is available. This program doesn't attempts to repair the filesystem, it only tries to pull files from damaged filesystems and copy them to a safe location. Also, the Btrfs filesystem checker (aka fsck) can now repair extent allocation tree corruptions (more repair modes in progress).

1.1.2. Btrfs: Metadata blocks bigger than 4KB

Btrfs was designed from the start to support blocks of multiple sizes, but the code wasn't ready and was disabled, so the Btrfs filesystems used as block size the size of a memory page (4KB in x86). In this version, support for metadata blocks bigger than a page size has been re-enabled, so Btrfs can use metadata blocks of up to 64KB in size (16/32KB seem to work better and are recommended). Support is enabled at mkfs time (f.e. mkfs.btrfs -l 32K). These block sizes cut down the size of the extent allocation tree dramatically and fragments much less. Code: [http://git.kernel.org/linus/727011e07cbdf87772fcc1999cccd15cc915eb62 (commit 1], [http://git.kernel.org/linus/b5d67f64f9bc656970dacba245410f0faedad18e 2)]

1.1.3. Btrfs: Performance improvements

Btrfs has improved the performance in several areas. The bigger metadata blocks alone give Btrfs a performance gain, as the extent allocation tree overhead and metadata fragmentation is greatly reduced. But there are other performance improvements: The way Btrfs works with the Linux page cache has been reworked and it's now much faster. CPU usage has been reduced. Also, the copy-on-write mechanisms didn't play well with the Linux VM and forced Btrfs to do many more reads than it should, further tuning has been done to prevent that.

As result of these improvements, the performance of metadata workloads is much faster. In a benchmark consisting in creating 32 million empty files, Btrfs created 170.000 files per second, whereas ext4 and XFS created 110.000 files/second and 115.000 files/second respectively. I/O graphs comparing Btrfs [https://plus.google.com/photos/105274031445877094012/albums/5725819775428547617/5725819775250665666 performance in 3.3] and [https://plus.google.com/photos/105274031445877094012/albums/5725820959365950049/5725821050029040946 performance in 3.4]. Code: [http://git.kernel.org/linus/3083ee2e18b701122a3b841db83448543a87a583 (commit)], [http://git.kernel.org/linus/5df4235ea15bd39f441ef334d8329b3d46b2cc57 (commit)], [http://git.kernel.org/linus/4f2de97acee6532b36dd6e995b858343771ad126 (commit)], [http://git.kernel.org/linus/cfed81a04eb555f5606d1b6a54bdbabab0ee1ac3 (commit)], [http://git.kernel.org/linus/0b32f4bbb423f02acee6d43cd442f5f0775db7e0 (commit)]

1.1.4. Btrfs: Better error handling

Many places of the Btrfs codebase weren't reliable (not because the data could be harmed, the filesystem is designed to keep the data always safe), but because many code functions didn't handle unexpected conditions, instead they would just stop the system by panic'ing it. In this version, Btrfs has been audited to handle these situations correctly: When one of those unexpected errors happens, the current transactions will be aborted, errors will be returned to the userspace callers, and the filesystem will enter in read-only mode, as it is the tradition in Linux. Code: [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=9613bebb223dea3179c265dc31e1bb41ae39f321 (commit)].

1.2. GPU drivers

1.2.1. GPU: Early support of Nvidia GeForce 600 'Kepler'

Nvidia announced new Kepler GPUs (GeForce 600 Series) on 22 March, and that was the day the Nouveau team asked to get basic modesetting support (no 3D, etc) for it merged in the main kernel. A quote from a Nouveau developer: "Its quite amazing that nouveau can support a GPU on its launch day even if its just unaccelerated modesetting". [http://nouveau.freedesktop.org/wiki/NVC0_Firmware External firmware] and updated graphic software stack are required. Code: [http://git.kernel.org/linus/ba331d5decbfe1cc8b1bf10fb7005f4b972c4f0e (commit)]

The Nouveau driver has also been "unstaged" and now it's considered ready for widespread use.

1.2.2. GPU: Support for AMD Radeon 7xxx and Trinity APU series

The newest GPU and APUs from AMD (Radeon 7xxx and Trinity APU series) are supported in this version. Code: [http://git.kernel.org/linus/09fa30226130652af75152d9010c603c66d46f6e (commit)]

1.2.3. GPU: Support of Intel Medfield graphics

This release adds experimental support for the GMA500 Medfield graphics. Medfield is a embedded architecture targeted for smartphones. Code: [http://git.kernel.org/linus/026abc333205c1fff80138b8c2cac3d0347685f4 (commit)]

1.3. New X32 ABI: 64-bit mode with 32-bit pointers

The 64 bits mode of x86 CPUs enlarges the CPU registers to 64 bit, allowing to address larger (>4GB) amounts of memory. This widening, however, has a drawback. Because memory addresses are 64-bit wide, pointers occupy 64 bits of space, the double of space used in 32 bits mode, so binaries compiled for the 64-bit mode are bigger, and when these programs run they use more RAM. And since they are bigger they can cause a performance loss, because with bigger memory addresses, less CPU instructions will fit in the CPU caches.

Some programs have workloads CPU and pointer intensive enough to care about this performance, but with memory requirements not big enough to care about 64-bit memory addressing. They can avoid the 64-bit pointer overhead by just using the 32 bits mode: Processors still allow to run 32-bit operative systems, or run 32-bit programs on top of 64-bit kernels. But this choice also has problems. When a program runs in 32-bit mode, it loses all the other features of the 64-bit mode: larger number of CPU registers, better floating-point performance, faster PIC (position-independent code) shared libraries, function parameters passed via registers, faster syscall instruction...

So a new X32 kernel ABI has been created. A program compiled for this new ABI can run in the 64-bit mode, with all its features, but it uses 32 bits pointers and 32-bit long C type. So applications who need it can enjoy the performance of the 64-bit mode, but with the memory requirements of a 32 bits ABI. Code: [http://git.kernel.org/linus/a591afc01d9e48affbacb365558a31e53c85af45 (commit)]

Recommended LWN article: [https://lwn.net/Articles/456731/ The x32 system call ABI]

Slides from the developers: [http://www.linuxplumbersconf.net/2011/ocw//system/presentations/531/original/x32-LPC-2011-0906.pptx link]

Official X32 coordination site: http://sites.google.com/site/x32abi

1.4. x86 CPU driver autoprobing

There's a growing number of drivers that support a specific x86 feature or CPU. Currently loading these drivers currently on a generic distribution requires various driver specific hacks and it often doesn't work. For example a common issue is not loading the SSE 4.2 accelerated CRC module: this can significantly lower the performance of Btrfs which relies on fast CRC. Another issue is loading the right CPUFREQ driver for the current CPU. Currently distributions often try all all possible driver until one sticks, which is not really a good way to do this.

Linux already has autoprobing mechanisms for drivers, based in kernel notifications and udev. In this release, Linux adds auto probing support for CPU drivers, based on the x86 CPUID information, in particular based on vendor/family/model number and also based on CPUID feature bits. Code: [http://git.kernel.org/linus/644e9cbbe3fc032cc92d0936057e166a994dc246 (commit 1)], [http://git.kernel.org/linus/3bd391f056df61e928de1680ff4a3e7e07e5b399 2], [http://git.kernel.org/linus/b66b8b9a4a79087dde1b358a016e5c8739ccf186 3], [http://git.kernel.org/linus/9061e0e16700ef228837e96987ff51794c956197 4], [http://git.kernel.org/linus/267fc9788d0cdb77edafb506063f06961e1418f5 5], [http://git.kernel.org/linus/9b38096fde5f9b93c3657911c3be7892cc155cbd 6], [http://git.kernel.org/linus/fa8031aefec0cf7ea6c2387c93610d99d9659aa2 7], [http://git.kernel.org/linus/78ff123b05fb15beb1ad670372eea0d299d0b8af 8], [http://git.kernel.org/linus/fad12ac8c8c2591c7f4e61d19b6a9d76cd49fafa 9)]

1.5. Verifiable boot path with the device mapper "verity" target

The device-mapper's "verity" target allows to use a device to store cryptographic hashes of the blocks of a filesystem. This device can be used to check every read attempt to the filesystem, and if the hash of the block doesn't match with the hash of the filesystem, the read fails. This target is used by products such as Chrome OS and Netflix to ensure that the operative system isn't modified, and it can also be used to boot from a known-good device (like a USB drive or CD).

Recommended LWN article: [https://lwn.net/Articles/459420/ dm-verity]

Code: [http://git.kernel.org/linus/a4ffc152198efba2ed9e6eac0eb97f17bfebce85 (commit)]

1.6. Support a external read-only device as origin source of a thin provisioned LVM volume

Device mapper supports thin provisioning (creation of filesystems larger than the total storage of the disks). Now, it also supports the use of an external read-only device as an origin for the thinly-provisioned volume. Any read to an unprovisioned area of the thin device will be passed through to the origin. Writes trigger the allocation of new blocks as usual.

One use case for this is VM hosts that want to run guests on thinly-provisioned volumes but have the base image on another device (possibly shared between many VMs).

Code: [http://git.kernel.org/linus/2dd9c257fbc243aa76ee6db0bb8371f9f74fad2d (commit)]

1.7. perf: GTK2 report GUI, better assembly visualization, branch profiling, filtering of users and threads

1.8. 'Yama' security module

Linux has several security modules: selinux, apparmor, etc. Yama is a new security module that collects a number of system-wide DAC security protections that are not handled by the core kernel itself. For now, Yama restricts the ptrace interface, which allow a process to examine the memory and running state of any of the processes of the same user.

Code: [http://git.kernel.org/linus/2d514487faf188938a4ee4fb3464eeecfbdcf8eb (commit)]

1.9. QNX6 filesystem

The qnx6fs is used by newer QNX operating system versions. (e.g. Neutrino). It got introduced in QNX 6.4.0 and is used default since 6.4.1. This release adds read-only support.

Code: [http://git.kernel.org/linus/5d026c7242201e7c9d0e12fcb2bcaffead9d59fd (commit)]

2. Driver and architecture-specific changes

All the driver and architecture-specific changes can be found in the [http://kernelnewbies.org/Linux_3.4_DriverArch Linux_3.4_DriverArch page]

3. File systems

4. Various core changes

5. Memory management

6. Networking

7. Virtualization

8. Crypto

9. Security

10. Block

11. Perf profiling


KernelNewbies: Linux_3.4 (last edited 2012-07-22 14:09:10 by diegocalleja)