KernelNewbies:

1. Prominent features

1.1. Bigger memory limits

Original x86-64 was limited by 4-level paging to 256 TiB of virtual address space and 64 TiB of physical address space. People are already bumping into this limit: some vendors offers servers with 64 TiB of memory today. To overcome the limitation upcoming hardware will introduce support for 5-level paging. It is a straight-forward extension of the current page table structures adding one more layer of translation. It bumps the limits to 128 PiB of virtual address space and 4 PiB of physical address space. This "ought to be enough for anybody" ©.

On x86, 5-level paging enables 56-bit userspace virtual address space. Not all user space is ready to handle wide addresses. It's known that at least some JIT compilers use higher bits in pointers. It collides with valid pointers with 5-level paging and leads to crashes. To mitigate this, the Linux kernel will not allocate virtual address space above 47-bit by default. Userspace can ask for allocation from full address space by specifying hint address above 47-bits.

Recommended LWN article: [https://lwn.net/Articles/717293/ Five-level page tables]

Code: [https://git.kernel.org/linus/ee00f4a32a76ef631394f31d5b6028d50462b357 commit], [https://git.kernel.org/linus/b569bab78d8df157a6f91070af827753e4d1787c commit], [https://git.kernel.org/linus/44b04912fa72489d403738f39e1c782614b7ae7c commit], [https://git.kernel.org/linus/77ef56e4f0fbb350d93289aa025c7d605af012d4 commit], [https://git.kernel.org/torvalds/c/b1b6f83ac938d176742c85757960dec2cf10e468 merge]

1.2. Add support for AMD Secure Memory Encryption

Secure Memory Encryption can be used to mark individual pages of memory as encrypted through the page tables. A page of memory that is marked encrypted will be automatically decrypted when read from DRAM and will be automatically encrypted when written to DRAM. Secure Memory Encryption can therefore be used to protect the contents of DRAM from physical attacks on the system.

Recommended LWN article: [https://lwn.net/Articles/686808/#sme Two approaches to x86 memory encryption]

AMD Memory encryption whitepaper: [http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_Memory_Encryption_Whitepaper_v7-Public.pdf link]

Code: [https://git.kernel.org/linus/c262f3b9a3246da87c66ce398cd7e30d8f1529ea commit], [https://git.kernel.org/linus/aac7b79eea6118dee3da9b99dcd564471672806d commit], [https://git.kernel.org/linus/f7750a79568788473c5e8092ee58a52248f34329 commit], [https://git.kernel.org/linus/872cbefd2d9c52bd0b1e2c7942c4369e98a5a5ae commit], [https://git.kernel.org/linus/9af9b94068fb1ea3206a700fc222075966fbef14 commit], [https://git.kernel.org/linus/7744ccdbc16f0ac4adae21b3678af93775b3a386 commit], [https://git.kernel.org/linus/33c2b803edd13487518a2c7d5002d84d7e9c878f commit], [https://git.kernel.org/linus/5868f3651fa0dff96a57f94d49247d3ef320ebe2 commit], [https://git.kernel.org/linus/fd7e315988b784509ba3f1b42f539bd0b1fca9bb commit], [https://git.kernel.org/linus/21729f81ce8ae76a6995681d40e16f7ce8075db4 commit], [https://git.kernel.org/linus/eef9c4abe77f55b1600f59d8ac5f1d953e2f5384 commit], [https://git.kernel.org/linus/f88a68facd9a15b94f8c195d9d2c0b30c76c595a commit], [https://git.kernel.org/linus/7f8b7e7f4ccbbd1fb8badddfabd28c955aea87b4 commit], [https://git.kernel.org/linus/b9d05200bc12444c7778a49c9694d8382ed06aa8 commit], [https://git.kernel.org/linus/d68baa3fa6e4d703fd0c7954ee5c739789e7242f commit], [https://git.kernel.org/linus/a19d66c56af1c52b8b463bf94d21116ae8c1aa5a commit], [https://git.kernel.org/linus/f99afd08a45fbbd9ce35a7624ffd1d850a1906c0 commit], [https://git.kernel.org/linus/38eecccdf488e38ee93690cfe9ec1914b73f512f commit], [https://git.kernel.org/linus/8f716c9b5febf6ed0f5fedb7c9407cd0c25b2796 commit], [https://git.kernel.org/linus/5997efb967565e858259401af394e8449629c1f0 commit], [https://git.kernel.org/linus/1de328628cd06b5efff9195b57bdc1a64680814d commit], [https://git.kernel.org/linus/77bd2342d4304bda7896c953d424d15deb314ca3 commit], [https://git.kernel.org/linus/163ea3c83aeeb3908a51162c79cb3a7c374d92b4 commit], [https://git.kernel.org/linus/c7753208a94c73d5beb1e4bd843081d6dc7d4678 commit], [https://git.kernel.org/linus/648babb7078c6310d2af5b8aa01f086030916968 commit], [https://git.kernel.org/linus/f655e6e6b992a2fb0d0334db2620607b98df39e7 commit], [https://git.kernel.org/linus/2543a786aa25258451f3418b87a038c7ddaa2e85 commit], [https://git.kernel.org/linus/46d010e04a637ca5bbdd0ff72554d9c06f2961c9 commit], [https://git.kernel.org/linus/95cf9264d5f36c291c1c50c00349f83348e6f9c7 commit], [https://git.kernel.org/linus/d0ec49d4de90806755e17289bd48464a1a515823 commit], [https://git.kernel.org/linus/bba4ed011a52d494aa7ef5e08cf226709bbf3f60 commit], [https://git.kernel.org/linus/f2f931c6819467af5260a21c59fb787ce2863f92 commit], [https://git.kernel.org/linus/8458bf94b0399cd1bca6c437366bcafb29c230c5 commit], [https://git.kernel.org/linus/db516997a985b461f021d594e78155bbc7fc3e7e commit], [https://git.kernel.org/linus/6ebcb060713f614c92216482eed501b31cee74ec commit], [https://git.kernel.org/linus/e505371dd83963caae1a37ead9524e8d997341be commit], [https://git.kernel.org/linus/7375ae3a0b79ea072f4c672039f08f5db633b9e1 commit], [https://git.kernel.org/linus/aca20d5462149333ba8b24a4a352be5b7a00dfd2 commit]

1.3. Better kernel traces with the ORC unwinder

This release includes a new "unwinder". An "unwinder" is what prints the list of functions (aka. stack trace, callgraph, call stack...) that have been executed before reaching a determinate point of the code, and it's used, for example, the list of functions that lead to a crash when a kernel oopses. The new unwinder is called ORC, an alias for "Oops Rewind Capability", and has been developed as an simpler alternative to the DWARF debuginfo format.

Linux already has an unwinder, and while it usually works well, it isn't reliable in all situations, which causes troubles for modern functionality like live patching that requires completely reliable stack traces. It also requires a functionality called "frame pointers" (CONFIG_FRAME_POINTERS) to print complete call stacks. Frame pointers make GCC add instrumentation code to every function in the kernel, which increases the size of the kernel executable code by about 3.2%, resulting in a broad kernel-wide slowdown, and more for some workloads. This option is enabled by default in some Linux distros.

In contrast, the ORC unwinder does not need to insert code anywhere so it has no effect on text size or runtime performance, because the debuginfo (about 2-4MiB) is placed out of band. So the ORC unwinder provides a nice performance improvement across the board compared with frame pointers, while at the same time having reliable stack traces.

Recommended LWN article: [https://lwn.net/Articles/728339/ The ORCs are coming]

Recommended article: [http://www.codeblueprint.co.uk/2017/07/31/the-orc-unwinder.html The Linux x86 ORC Stack Unwinder]

Code: [https://git.kernel.org/linus/ee9f8fce99640811b2b8e79d0d1dbe8bab69ba67 commit], [https://git.kernel.org/linus/39358a033b2e4432052265c1fa0f36f572d8cfb5 commit]

1.4. Better cpufreq coordination with SMP

In Linux, notifications of task scheduler events are sent to the cpufreq subsystem, so that it can increase the frequency if needed, and achieve good interactivity. However, the cpufreq drivers are not called when the events are happening in different CPUs, for example, a new process being created in another CPU. This release makes task scheduler to update the cpufreq policies for remote CPUs as well. The schedutil, ondemand and conservative governors are updated to process cpufreq updates for remote CPUs (the intel_pstate driver is updated to always reject them).

Recommended LWN article: [https://lwn.net/Articles/732740/ CPU frequency governors and remote callbacks]

Code: [https://git.kernel.org/linus/674e75411fc260b0d4532701228cfe12fc090da8 commit], [https://git.kernel.org/linus/99d14d0e16fadb4de001bacc0ac0786a9ebecf2a commit]

1.5. Faster TBL flushing on newer Intel CPUs

PCID is a hardware feature on newer Intel CPUs that attaches an address space tag to TLB entries and thus allows to skip TLB flushing in many cases, even if we switch mm's.

2. Core (various)

3. File systems

4. Memory management

5. Block layer

6. Tracing, perf and BPF

type that holds references to sock structs. Then with a new sk redirect bpf helper BPF programs can use the map to redirect skbs between sockets. To use this infrastructure a new BPF program BPF_PROG_TYPE_SK_SKB is added that allows users to reference sock details, such as port and ip address fields, to build useful socket layer program [https://git.kernel.org/linus/f26de110f4f1d8d2490d1f9f003d5abbde030f03 commit], [https://git.kernel.org/linus/45f91bdcd5c5ba559a4bb7c3a0e0709476cf570f commit], [https://git.kernel.org/linus/db5980d804d7158917ad4b9186c78b2a3f1db4ef commit], [https://git.kernel.org/linus/b005fd189cec9407b700599e1e80e0552446ee79 commit], [https://git.kernel.org/linus/a6f6df69c48b86cd84f36c70593eb4968fceb34a commit], [https://git.kernel.org/linus/174a79ff9515f400b9a6115643dafd62a635b7e6 commit], [https://git.kernel.org/linus/8a31db5615667956c513d205cfb06885c3ec6d0b commit], [https://git.kernel.org/linus/69e8cc134bcbf0ccfcf852c400b8e6788d1d0038 commit], [https://git.kernel.org/linus/41bc94f535ef454e325a6d4db085ec345376de6c commit], [https://git.kernel.org/linus/6f6d33f3b3d0f53799d120d28abd13ad90041549 commit], [https://git.kernel.org/linus/464bc0fd6273d518aee79fbd37211dd9bc35d863 commit], [https://git.kernel.org/linus/2f857d04601a1bb56958b95a9f180bce0e91e5e6 commit], [https://git.kernel.org/linus/d26e597d87635d90128fafb3f6bb0a14d972d952 commit], [https://git.kernel.org/linus/6fd28865c2a7e5ea12cb1f7ef3edee5a2042905e commit], [https://git.kernel.org/linus/ed85054d34e2dfb5e9fac95980cf038ecf19225c commit], [https://git.kernel.org/linus/81374aaa2693f8d3cd6cf3656a02ac8cf5c7ebea commit], [https://git.kernel.org/linus/78aeaaef997db7096a17d0d3572a7940ffa5c9a0 commit], [https://git.kernel.org/linus/08848246639218ae58acdf3321bc7b693062f31c commit], [https://git.kernel.org/linus/3f0d6a16989da252d4014c3fb7334369c891f91e commit], [https://git.kernel.org/linus/5a67da2a71c64daeb456f6f3e87b5c7cecdc5ffa commit]

7. Virtualization

8. Security

9. Networking

10. Architectures

11. List of merges

12. Other news sites

KernelNewbies: Linux_4.14 (last edited 2017-11-12 19:26:26 by diegocalleja)