Attachment 'hacking_the_wholism_of_linux_net.txt'
Download
Toggle line numbers
1 root@slackware-13.1:/home/forsaken# uname -a
2 Linux common-slackware 2.6.33.4-smp #2 SMP Wed May 12 22:47:36 2044
3 i686 Intel(R) Core(TM)2 Duo CPU E7500 @ 2.93GHz GenuineIntel
4 GNU/Linux
5
6
7 |=-----------------------------------------------------------------=|
8 |=------------=[Hacking the wholism of GNU/Linux net*]=------------=|
9 |=----------------=[ Netfilter <====> NIC driver ]=----------------=|
10 |=-----------------------------------------------------------------=|
11 |=-----------------------------------------------------------------=|
12 |=-------------------=[ By Shawn the R0ck ]=---------------------=|
13 |=-------------------=[ <citypw@gmail.com> ]=---------------------=|
14 |=-----------------------------------------------------------------=|
15 |=------------------------=[ July 7 2011 ]=-----------------------=|
16 |=-----------------------------------------------------------------=|
17
18
19 --[ Contents
20
21 0. Introduction
22
23 1. What is iptables/Netfilter framework
24
25 1.1 Hook your packet
26
27 1.1.1 Iptables sample
28
29 1.2 Write your own hook function
30
31 1.3 What can Netfilter do
32
33 1.3.1 Other important components in Netfilter
34
35 2. Linux networking systems: A monkey-coder's perspective
36
37 2.1 Initialization of the NIC driver
38
39 2.2 RX packets
40
41 2.3 TX packets
42
43 2.4 Three ways of packet's traveling
44
45 2.4.1 Network to Host
46
47 2.4.2 Network to Network
48
49 2.4.3 Host to Network
50
51 2.5 A bigger picture
52
53 3. Conclusion
54
55 3.1 Gratitude
56
57 4. References
58
59
60
61 --[ 0. Introduction
62
63 This article will discuss 2 topics of GNU/Linux networking system at
64 the introduction-level. In the 1st part I will give you a simple way
65 to introduce Netfiler and iptables. I'm not going further into
66 userspace tools and just give some cmd-lines of iptables as examples
67 that you can understand the relationship between iptables and
68 netfilter. I will also provide some source codes which the original
69 ones come from one of the great papers in phrack magazine[1]. You can
70 see bunch of encoded stuff at bottom of this article, but you must try
71 to use the tools "uudecode"(Phrack guys would not tell you this
72 because they treated you as hackers || I'm not implying that I treat
73 you as "users" -_-) to get source code of the netfilter samples which
74 tested in GNU/Linux kernel 2.6.33[2].
75
76 In the 2nd part of this article I will talk about how a packet travels
77 around in GNU/Linux kernel from NIC driver layer to network stack and
78 use the REALTEK 8169 NIC driver source code as example, which can be
79 found in /usr/src/linux-2.6.33.4/drivers/net/r8169.c.
80
81 I will follow the principle "read the fucking source code" in
82 discussion.
83
84 I have been using GNU/Linux for 4 years. And I started hacking on
85 network system( both of user space and kernel space) of GNU/Linux one
86 year ago. So I'm trying to make the pieces of my notes into one
87 article you are reading now. Share the information, be free and
88 opening. "By the community, for the community." should be a hacker's
89 creed.
90
91
92 --[ 1. What is iptables/netfilter framework
93
94 Netfilter is a framework that provides hook mechanism for those who
95 need to write their own functions for mangling packets within
96 GNU/Linux kernel. Iptables is the userspace command-line program to
97 configure the filtering rules of the GNU/linux kernel. Both of them
98 are free softwares.
99
100
101 ----[ 1.1 Hook your packet
102
103 Netfilter can filter the packet of the kernel network stack by
104 inserting your own kernel modules. Netfilter has 4
105 tables(filter,nat,mangle,raw) and 5 chains. The hooks is the
106 implementation of chains in protocol stack. The different protocol
107 families(IPv4, IPv6, etc) of hooks linked each other by linked
108 list. Every table may have multiple policies stored in an array.
109
110 The declaration of the symbols can be found in
111 /usr/src/linux-2.6.33.4/include/linux. These hooks are displayed in
112 the table below:
113
114 Table 1: Available IPv4 hooks
115
116 Hook Called
117 NF_IP_PRE_ROUTING After sanity checks, before routing decisions.
118 invoked in ip_rcv().
119
120 NF_IP_LOCAL_IN After routing decisions if packet is for this host.
121 invoked in ip_local_deliver().
122
123 NF_IP_FORWARD If the packet is destined for another interface.
124 invoked in ip_forward().
125
126 NF_IP_LOCAL_OUT For packets coming from local processes on their way out.
127 invoked in __ip_local_out().
128
129 NF_IP_POST_ROUTING Just before outbound packets "hit the wire".
130 invoked in ip_output().
131
132
133 The NF_IP_PRE_ROUTING hook is the first one that will be invoked when
134 a packet arrive. The different tables will invoke different functions
135 for different hooks. Finnaly, each function has to deal with the
136 policy matched by invoking ipt_do_tables() which can be found in
137 /usr/src/linux-2.6.33.4/net/ipv4/netfilter/ip_tables.c. Netfilter
138 defined 3 default tables for different uses. You can see the Table
139 below:
140
141 Table 2: Tables, Hooks and Policy
142
143 *-------------------------------------------------------------------------------------------*
144 |-[Table Name]-|----[ Hook Name]----|---[ Policy Functions]---|----[ Description ]----------|
145 | | | linux-2.6.33.4/net/ipv4/netfilter/nf_nat_standalone.c |
146 | | NF_IP_PRE_ROUTING | nf_nat_in | Translation work of ip |
147 |----=[nat]=---| NF_IP_POST_ROUTING | nf_nat_out | address and network port |
148 | | NF_IP_LOCAL_OUT | nf_nat_local_in | |
149 |-------------------------------------------------------------------------------------------|
150 | | | linux-2.6.33.4/net/ipv4/netfilter/iptable_filter.c |
151 | | NF_IP_LOCAL_IN | ipt_local_in_hook | Access control for packet |
152 |--=[filter]=--| NF_IP_FORWARD | ipt_hook | |
153 | | NF_IP_LOCAL_OUT | ip_local_out_hook | |
154 --------------------------------------------------------------------------------------------|
155 | | | linux-2.6.33.4/net/ipv4/netfilter/iptable_mangle.c |
156 | | NF_IP_PRE_ROUTING | ipt_pre_routing_hook | |
157 |--=[mangle]=--| NF_IP_LOCAL_IN | ipt_local_in_hook | Tagging the packet to |
158 | | NF_IP_FORWARD | ipt_forward_hook | mangling the options like |
159 | | NF_IP_LOCAL_OUT | ipt_local_hook | TTL, TOS, etc |
160 | | NF_IP_POST_ROUTING | ipt_post_routing_hook | |
161 *-------------------------------------------------------------------------------------------*
162
163 the filter table Description: What's the most important feature of
164 Netfilter? Filtering the packet. That's why this table is the most
165 important. If you want a effective filter rule, insert it to the
166 first.
167
168 the nat table Description: It does exist in 3 chains. The
169 implementation of Netfilter's nat table is based on connection
170 tracking for the supporting source NAT, Destination NAT and some
171 address translation mode including 1-to-1, many-to-1,
172 many-to-many. Because it's based on connection tracking, so the nat
173 table would only process new/related packet. The other state's packet
174 will direct to address translation according to NAT information of
175 connection tracking. NAT uses different methods to deal with different
176 protocols. NAT would only modify a few of source IP, destination IP,
177 source port or destination port if the packet is tcp or udp. But it
178 will modify the segments of id, type or code if the packet is icmp.
179
180 the mangle table's Description: It does exist in 5 chains. Netfilter
181 does not use the table mangle usually. The mangle table is used to
182 modify the TTL, TOS or tagging the mark for packet. TTL is used to
183 calculate how many routers the packet will pass in transportation. TOS
184 decides the priority of the packet. Tagging mark is used to deal with
185 policy route when you have more than 1 ISP wired in.
186
187 The process of a packet traversing the Netfilter is displayed in the
188 Figure below:
189
190 Figure 1: Traverse the Netfilter
191
192 +--------------+
193 /->| local socket |--\
194 User space / +--------------+ \
195 ------------------------/----------------------------\----------------------------
196 Kernel space / |
197 | \*/
198 +----------------+ +-----------------+
199 | NF_IP_LOCAL_IN | | NF_IP_LOCAL_OUT |
200 +----------------+ +-----------------+
201 /*\ |
202 | |
203 packet-in | |
204 *-------------------* $--------$ | packet-out
205 -->| SNAT | | route | +---------------+ \*/ *-------------*
206 | NF_IP_PRE_ROUTING | --->| decsion|--->| NF_IP_FORWARD | ----->| DNAT |--->
207 *-------------------* $--------$ +---------------+ | POSTROUTING |
208 *-------------*
209
210
211
212 The hook functions will return some values to tell Netfilter what to
213 do then, when the hook functions are done. These values are displayed
214 in the Table below:
215
216 Table 3: Return code of hook function
217
218 Return Code Meaning
219 NF_DROP Discard the packet.
220 NF_ACCEPT Keep the packet.
221 NF_STOLEN Forget about the packet.
222 NF_QUEUE Queue packet for userspace.
223 NF_REPEAT Call this hook function again.
224
225 You can see the description of Bioforge's article[1] about these
226 return value:
227
228 "The NF_DROP return code means that this packet should be dropped
229 completely and any resources allocated for it should be
230 released. NF_ACCEPT tells Netfilter that so far the packet is still
231 acceptable and that it should move to the next stage of the network
232 stack. NF_STOLEN is an interesting one because it tells Netfilter to
233 "forget" about the packet. What this tells Netfilter is that the hook
234 function will take processing of this packet from here and that
235 Netfilter should drop all processing of it. This does not mean,
236 however, that resources for the packet are released. The packet and
237 it's respective sk_buff structure are still valid, it's just that the
238 hook function has taken ownership of the packet away from Netfilter.
239 Unfortunately I'm not exactly clear on what NF_QUEUE really does so
240 for now I won't discuss it. The last return value, NF_REPEAT requests
241 that Netfilter calls the hook function again. Obviously one must be
242 careful using NF_REPEAT so as to avoid an endless loop."
243
244 The netfilter will send the packets to the userspace programs(such as
245 Snort) after NF_QUEUE return.
246
247 ------[ 1.1.1 Iptables samples
248
249 Iptables is a user-space tool that you can use it for
250 adding/removing/modifying firewall rules. As I said in the beginning
251 of this article, I'm not going to dig deeper into it. Read your "man
252 iptables" if you want details of how to use.
253
254 Case 1: Append a rule to the NF_IP_LOCAL_IN hook of the filter table,
255 which the rule is to drop all packets that source IP address is
256 192.168.0.10 trying to pass to the Host. Then list the filter table's
257 rules.
258
259 root@slackware-13.1:/home/forsaken# iptables -A INPUT -s 192.168.0.10 -j DROP
260 root@slackware-13.1:/home/forsaken# iptables -L
261 Chain INPUT (policy ACCEPT)
262 target prot opt source destination
263 DROP all -- 192.168.0.10 anywhere
264
265 Chain FORWARD (policy ACCEPT)
266 target prot opt source destination
267
268 Chain OUTPUT (policy ACCEPT)
269 target prot opt source destination
270
271
272 Case 2: Insert a rule to the NF_IP_POST_ROUTING hook of the mangle
273 table, which the rule is to drop all packets that destination IP
274 address is 192.168.0.10.
275
276 root@slackware-13.1:/home/forsaken# iptables -t mangle -F
277 root@slackware-13.1:/home/forsaken# iptables -t mangle -I POSTROUTING -d 192.168.0.10 -j DROP
278 root@slackware-13.1:/home/forsaken# iptables -t mangle -L
279 Chain PREROUTING (policy ACCEPT)
280 target prot opt source destination
281
282 Chain INPUT (policy ACCEPT)
283 target prot opt source destination
284
285 Chain FORWARD (policy ACCEPT)
286 target prot opt source destination
287
288 Chain OUTPUT (policy ACCEPT)
289 target prot opt source destination
290
291 Chain POSTROUTING (policy ACCEPT)
292 target prot opt source destination
293 DROP all -- anywhere slackware-13.1.org
294
295
296 ----[ 1.2 Write your own hook function
297
298 Before invoking the function nf_register_hook(), we need to declare a
299 structure and initialize it. The declaration of the structure is
300 nf_hook_ops which can be found it in
301 /usr/src/linux-2.6.33.4/include/linux/netfilter.h.
302
303 struct nf_hook_ops {
304 struct list_head list;
305
306 /* a pointer to function */
307 nf_hookfn *hook;
308 struct module *owner;
309 /* protocol family, we use IPv4 in case */
310 u_int8_t pf;
311 /* which hook point we hook up */
312 unsigned int hooknum;
313 /* Hooks are ordered in ascending priority. */
314 int priority;
315 };
316
317 Your hook function's prototype is like below:
318
319 typedef unsigned int nf_hookfn(unsigned int hooknum,
320 struct sk_buff *skb,
321 const struct net_device *in,
322 const struct net_device *out,
323 int (*okfn)(struct sk_buff *));
324
325
326 Let's read the source code to initialize the structure. This snippet
327 code is part of our source code samples. You might be interested in
328 reading the complete one. It's quiet easy to understand!
329
330 static int init_filter_if()
331 {
332 printk("initializing the hooks!\n");
333
334 /* remember which hook you specified */
335 nfho.hook = check_tcp_packet;
336 nfho.hooknum = NF_IP_PRE_ROUTING;
337 nfho.pf = PF_INET; /* ipv4 protocols */
338 nfho.priority =NF_IP_PRI_FIRST;
339
340 nf_register_hook(&nfho);
341
342 return 0;
343 }
344
345
346 ----[ 1.3 What can Netfilter do
347
348 Netfilter can do a lot of hacks that will depend on how brilliant
349 ideas you have. You can do:
350
351 ---> Implementation of firewall, eg: netfilter/iptables[4] are the best case.
352 ---> Implementation of KIDS[5]
353 ---> Protocol-based and application-based systems, eg: Bioforge's ftp-sniffer is a good example.
354
355 I'm a man who is lack of creative imagination. So I just listed these
356 I knew. I believe you can do more hacks on Netfilter-_-
357
358
359 ------[ 1.3.1 Other important components in Netfilter
360
361 Of course, connection tracking is one of the important components in
362 Netfilter framework. Connection tracking provides a kind of mechanism
363 to track the network connections. Connection tracking is the key of
364 implementation of the NAT and stateful firewall. The connection state
365 is completely independent of any upper-level state, such as TCP's
366 state. Because connection tracking only concerns about packets which
367 are passing the hooks of PREROUTING and POSTROUTING. Netfilter
368 connection can be manipulated by the user-space tool "conntrack" and
369 be used of checking the states with Iptables. Here I list some common
370 states (Referenced from Wikipedia):
371
372 *-------------------------------------------------------------------------------*
373 |NEW | trying to create a new connection |
374 |-------------------------------------------------------------------------------|
375 |ESTABLISHED | part of an already-existing connection |
376 |-------------------------------------------------------------------------------|
377 | | assigned to a packet that is initiating a new connection and |
378 | RELATED | which has been "expected". The aforementioned mini-ALGs set up |
379 | | these expectations, for example, when the nf_conntrack_ftp |
380 | | module sees an FTP "PASV" command. |
381 |-------------------------------------------------------------------------------|
382 |INVALID | the packet was found to be invalid, e.g. it would not adhere |
383 | | to the TCP state diagram. |
384 |-------------------------------------------------------------------------------|
385 |UNTRACKED | is a special state that can be assigned by the administrator to |
386 | | bypass connection tracking for a particular packet (raw table) |
387 *-------------------------------------------------------------------------------*
388
389
390 --[ 2. Linux networking systems: A monkey-coder's perspective
391
392 This part will discuss the linux kernel's network sub-systems
393 including the NIC driver's initialization, delivery/receipt of
394 packets, IP packet's processing.
395
396
397 ----[ 2.1 Initialization of the NIC driver
398
399 As ELDD[6] said, "NIC drivers are different from other driver classes
400 in that they do not rely on /dev or /sys to communicate with user
401 space. Rather, applications interact with a NIC driver via a network
402 interface (for example, eth0 for the first Ethernet interface) that
403 abstracts an underlying protocol stack.". By using the userspace tool
404 "ifconfig" can manipulate the NIC driver, which provides a set of
405 interfaces to communicate with NIC hardware (eg: rtl8169_open will be
406 invoked after running the command "ifconfig eth0 up").
407
408 Network Interface Cards usually are treated as the PCI (or USB in a
409 few cases) device objects in the linux kernel. There's a simple way to
410 understand the process of initialization of the RealTek 8169 NIC
411 driver that is displayed in the figure below:
412
413 Figure 2: Initialization of the NIC driver
414
415 +---------------------------+
416 | Loading the driver module |
417 +---------------------------+
418 |
419 |
420 \*/ *------*
421 +-----------------------+ | exit |
422 | rtl8169_init_module() | *------*
423 +-----------------------+ /|\
424 | |
425 | | N
426 \*/ |
427 +-----------------------+ +-------------------+ $=============$
428 | pci_register_driver() |--->| driver_register() |--->$ driver_find $
429 +-----------------------+ +-------------------+ $=============$
430 |
431 |
432 \|/
433 +------------------+
434 | bus_add_driver() |
435 +------------------+
436 |
437 \|/
438 +-----------------+
439 | driver_attach() |
440 +-----------------+
441 Iteration |
442 +--------------------+ |
443 | __driver_attach() | |
444 +--------------------+ |
445 | /|\ \|/
446 | | $====================$
447 | +---------$ bus_for_each_dev() $
448 \|/ $====================$
449 +--------------------+
450 | driver_probe_dev() |
451 +--------------------+
452 |
453 |
454 \|/
455 +--------------------+
456 | pci_device_probe() |
457 +--------------------+
458 |
459 |
460 \|/
461 +----------------------+ +------------------+ +-------------------+
462 | __pci_device_probe() |--->| pci_call_probe() |--->| local_pci_probe() |
463 +----------------------+ +------------------+ +-------------------+
464 |
465 |
466 \|/
467 *--------------------*
468 | rtl8169_init_one() |
469 *--------------------*
470
471 I suggest that you should read the kernel's Documentation[7], while
472 looking into the source code. The entry point and exit point of the
473 implementation in rtl8169 dirver are rtl8169_init_module() and
474 rtl8169_cleanup_module() defined in the source code:
475
476 module_init(rtl8169_init_module);
477 module_exit(rtl8169_cleanup_module);
478
479 The rtl8169_init_module() function will be invoked after "insmod" your
480 driver module. And you can see the function is only doing one thing:
481
482 static int __init rtl8169_init_module(void)
483 {
484 return pci_register_driver(&rtl8169_pci_driver);
485 }
486
487 The argument "rtl8169_pci_driver" is a structure of the pci_driver
488 which can be found in src/include/linux/pci.h
489
490 struct pci_driver {
491 struct list_head node;
492 char *name;
493 const struct pci_device_id *id_table; /* must be non-NULL for probe to be called */
494 int (*probe) (struct pci_dev *dev, const struct pci_device_id *id); /* New device inserted */
495 void (*remove) (struct pci_dev *dev); /* Device removed (NULL if not a hot-plug capable driver) */
496 int (*suspend) (struct pci_dev *dev, pm_message_t state); /* Device suspended */
497 int (*suspend_late) (struct pci_dev *dev, pm_message_t state);
498 int (*resume_early) (struct pci_dev *dev);
499 int (*resume) (struct pci_dev *dev); /* Device woken up */
500 void (*shutdown) (struct pci_dev *dev);
501 struct pci_error_handlers *err_handler;
502 struct device_driver driver;
503 struct pci_dynids dynids;
504 };
505
506 As you see the above structure contains some function pointers to your
507 own implementations for the NIC drivers. Let's see the rtl8169
508 driver's initialization for the structure:
509
510 static struct pci_driver rtl8169_pci_driver = {
511 .name = MODULENAME,
512 .id_table = rtl8169_pci_tbl,
513 .probe = rtl8169_init_one,
514 .remove = __devexit_p(rtl8169_remove_one),
515 .shutdown = rtl_shutdown,
516 .driver.pm = RTL8169_PM_OPS,
517 };
518
519 The information of the id table "rtl8169_pci_tbl" is related with the
520 implemenation of PCI architecture in linux kernel. And the prototype
521 of the rtl8169_init_one() function is displayed below:
522
523 static int __devinit
524 rtl8169_init_one(struct pci_dev *pdev, const struct pci_device_id *ent);
525
526 Where did you get these 2 parameters? Don't forget which function is
527 the caller: As you can see in above figure, it's
528 local_pci_probe(). These 2 parameters are created in PCI bus
529 enumerations.
530
531 static long local_pci_probe(void *_ddi)
532 {
533 struct drv_dev_and_id *ddi = _ddi;
534
535 return ddi->drv->probe(ddi->dev, ddi->id);
536 }
537
538 The declaration of the _ddi's structure can be found in
539 src/drivers/pci/pci-driver.c
540
541 struct drv_dev_and_id {
542 struct pci_driver *drv;
543 struct pci_dev *dev;
544 const struct pci_device_id *id;
545 };
546
547 The driver will do a lot of things in rtl8169_init_one(), such as
548 memory mapping, allocation for network device, setting up DMA,
549 etc. But We have to care about one line of source in
550 rtl8160_init_one():
551
552 netif_napi_add(dev, &tp->napi, rtl8169_poll, R8169_NAPI_WEIGHT);
553
554 This one is about the softirq we will talk about it later.
555
556
557 ----[ 2.2 RX packets
558
559 While you are trying to turn on (eg: ifconfig eth0 up) a Ethernet
560 interface, it will try to register a interrupt number by invoking
561 request_irq() in the rtl8169_open() funcion:
562
563 retval = request_irq(dev->irq, rtl8169_interrupt, (tp->features &
564 RTL_FEATURE_MSI) ? 0 : IRQF_SHARED,
565 dev->name, dev);
566
567 The NIC hardware will rise a interrupt to CPU when the NIC recieves
568 packets from network and then linux kernel start to execute the
569 interrupt handler (we use rtl8169_interrupt() in case) for processing
570 packets. In this article, we need to know a little concepts of
571 hardware interrupt. The other work flow of hardware interrupt is a
572 tough topic that beyond the range of this article. There is another
573 great paper[8] from phrack, which is worth reading.
574
575 After initializing the IDT (Interrupt Descriptor Table) in the kernel
576 booting stage. When a hardware device raises an interrupt to CPU, the
577 assembly code will execute at first, then it will jump to the familiar
578 C code function do_IRQ() which can be found in
579 src/arch/x86/kernel/entry_32.S:
580
581 /*
582 * Build the entry stubs and pointer table with some assembler magic.
583 * We pack 7 stubs into a single 32-byte chunk, which will fit in a
584 * single cache line on all modern x86 implementations.
585 */
586 .section .init.rodata,"a"
587 ENTRY(interrupt)
588 .text
589 .p2align 5
590 .p2align CONFIG_X86_L1_CACHE_SHIFT
591 ENTRY(irq_entries_start)
592 RING0_INT_FRAME
593 vector=FIRST_EXTERNAL_VECTOR
594 .rept (NR_VECTORS-FIRST_EXTERNAL_VECTOR+6)/7
595 .balign 32
596 .rept 7
597 .if vector < NR_VECTORS
598 .if vector <> FIRST_EXTERNAL_VECTOR
599 CFI_ADJUST_CFA_OFFSET -4
600 .endif
601 1: pushl $(~vector+0x80) /* Note: always in signed byte range */
602 CFI_ADJUST_CFA_OFFSET 4
603 .if ((vector-FIRST_EXTERNAL_VECTOR)%7) <> 6
604 jmp 2f
605 .endif
606 .previous
607 .long 1b
608 .text
609 vector=vector+1
610 .endif
611 .endr
612 2: jmp common_interrupt
613 .endr
614 END(irq_entries_start)
615
616 See, the common code starts at label common_interrupt and consists of
617 the following assembly language macros and instructions:
618
619 /*
620 * the CPU automatically disables interrupts when executing an IRQ vector,
621 * so IRQ-flags tracing has to follow that:
622 */
623 .p2align CONFIG_X86_L1_CACHE_SHIFT
624 common_interrupt:
625 addl $-0x80,(%esp) /* Adjust vector into the [-256,-1] range */
626 SAVE_ALL
627 TRACE_IRQS_OFF
628 movl %esp,%eax
629 call do_IRQ
630 jmp ret_from_intr
631 ENDPROC(common_interrupt)
632 CFI_ENDPROC
633
634 Then, we get to the C code do_IRQ() function after executing "call
635 do_IRQ". By invoking the fuc handle_irq() in do_IRQ() which can be
636 found in src/arch/x86/kernel/irq.c:
637
638 /*
639 * do_IRQ handles all normal device IRQ's (the special
640 * SMP cross-CPU interrupts have their own specific
641 * handlers).
642 */
643 unsigned int __irq_entry do_IRQ(struct pt_regs *regs)
644 {
645 struct pt_regs *old_regs = set_irq_regs(regs);
646
647 /* high bit used in ret_from_ code */
648 unsigned vector = ~regs->orig_ax;
649 unsigned irq;
650
651 exit_idle();
652 irq_enter();
653
654 irq = __get_cpu_var(vector_irq)[vector];
655
656 if (!handle_irq(irq, regs)) {
657 ack_APIC_irq();
658
659 if (printk_ratelimit())
660 pr_emerg("%s: %d.%d No irq handler for vector (irq %d)\n",
661 __func__, smp_processor_id(), vector, irq);
662 }
663
664 irq_exit();
665
666 set_irq_regs(old_regs);
667 return 1;
668 }
669
670 Finally, the handle_irq() function calls the rtl8169_interrupt()
671 function by a function pointer "desc->handle_irq(irq, desc)", which
672 can be found in src/arch/x86/kernel/irq_32.c. Now the packet got into
673 the interrupt implementation of the NIC driver:
674
675 bool handle_irq(unsigned irq, struct pt_regs *regs)
676 {
677 struct irq_desc *desc;
678 int overflow;
679
680 overflow = check_stack_overflow();
681
682 desc = irq_to_desc(irq);
683 if (unlikely(!desc))
684 return false;
685
686 if (!execute_on_irq_stack(overflow, desc, irq)) {
687 if (unlikely(overflow))
688 print_stack_overflow();
689 desc->handle_irq(irq, desc);
690 }
691
692 return true;
693 }
694
695
696 Many NIC drivers now are using NAPI's strategy that uses polling mode
697 while many hardware interrupts are rarising in period of time, and
698 then turn back to interrupt mode when not many packets need
699 processing. This is the best solution to avoid the large number of
700 hardware interrupts which might exhaust the CPU. There's a few steps
701 to go through with it:
702
703 1, In interrupt mode, the interrupt handler rtl8169_interrupt() posts
704 receive packets to protocol layers by scheduling NET_RX_SOFTIRQ:
705
706 if (likely(napi_schedule_prep(&tp->napi)))
707 __napi_schedule(&tp->napi);
708
709 It then disables NIC intetrrupts and switches to polling mode by
710 invoking __napi_schedule() to add the devices to a poll list:
711
712 /**
713 * __napi_schedule - schedule for receive
714 * @n: entry to schedule
715 *
716 * The entry's receive function will be scheduled to run
717 */
718 void __napi_schedule(struct napi_struct *n)
719 {
720 unsigned long flags;
721
722 local_irq_save(flags);
723 list_add_tail(&n->poll_list, &__get_cpu_var(softnet_data).poll_list);
724 __raise_softirq_irqoff(NET_RX_SOFTIRQ);
725 local_irq_restore(flags);
726 }
727
728 Both receipt and transmission methods of softirqs are registered in
729 net_dev_init() which can be found in src/net/core/dev.c:
730
731 open_softirq(NET_TX_SOFTIRQ, net_tx_action);
732 open_softirq(NET_RX_SOFTIRQ, net_rx_action);
733
734 2, By invoking rtl8169_poll() in the net_rx_action() function which
735 can be found in src/net/core/dev.c:
736
737 if (test_bit(NAPI_STATE_SCHED, &n->state)) {
738 work = n->poll(n, weight);
739 trace_napi_poll(n);
740 }
741
742 3, In the polling mode, the rtl8169_poll() processes packets in the
743 ingress queue. When the queue becomes empty, the driver re-enables
744 interrupts and switches back to interrupt mode by calling
745 napi_complete():
746
747 if (unlikely(work == weight)) {
748 if (unlikely(napi_disable_pending(n))) {
749 local_irq_enable();
750 napi_complete(n);
751 local_irq_disable();
752 } else
753 list_move_tail(&n->poll_list, list);
754 }
755
756
757 Figure 3: The process of RX packets
758
759 ---------------------------------------------------------------------------------------------------------------------------
760 +-----+ |
761 | NIC | |
762 +-----+ |
763 | |
764 | Raise a interrupt |
765 \|/ |
766 *-----------------------* |
767 | CPU-1 | CPU-2 | CPU-N | |
768 *-----------------------* |
769 | |
770 | Interrupt[n] |
771 \|/ | *---------------------*
772 +-----------+ | | softnet_data[CPU-n] |-----------+
773 | do_IRQ(n) | | *---------------------* |
774 +-----------+ | /*\ |
775 | | | add to poll list |
776 \|/ | | | raise softirq
777 +--------------+ +---------------------+ | +-------------------+ |
778 | handle_irq() |--->| rtl8169_interrupt() |----> | __napi_schedule() | |
779 +--------------+ +---------------------+ | +-------------------+ |
780 | \|/
781 | +-----------------+
782 | | net_rx_action() |
783 | +-----------------+
784 $================$ | $==============$ |
785 $ Interrupt mode $ | $ Polling mode $ |
786 $================$ | $==============$ \|/
787 /*\ | +----------------+
788 | | | rtl8169_poll() |
789 | | +----------------+
790 | | |
791 | | Work of RX |
792 | | \|/
793 | | +------------------+
794 | | +---------------| eth_type_trans() |
795 | | | +------------------+
796 | | \|/
797 | | $===========$ Y +-----------------------+
798 | | $ IS_VLAN $------>| rtl8169_rx_vlan_skb() |
799 | | $===========$ +-----------------------+
800 | | | |
801 | | N | |
802 | | \|/ |
803 | | +---------------------+ |
804 | | | netif_receive_skb() |---------->|
805 | | +---------------------+ |
806 | |
807 | \|/
808 | Yes, done the packet processing $-----------------$
809 +------------------------------------------------------| napi_complete() |
810 $-----------------$
811
812
813 There will be 5 steps to start running when the packet travels to
814 rtl8169_rx_interrupt:
815
816 1, netdev_alloc_skb() in rtl8169_alloc_rx_skb(), allocate a
817 receive buffer
818
819 2, skb_reserve(), add a 2-byte padding between the start of the packet
820 buffer and the beginning of the payload for align with IP header which
821 is 16-byte.
822
823 3, NIC hardware maps a memory space from DMA to memory. Copy the data
824 in DMA into a preallocated sk_buff when the data arrives.
825
826 4, skb_put(), extend the used data area of the buffer.
827
828 5, netif_receive_skb(), enqueue the packet for upper protocols/levels
829 to process.
830
831
832 ----[ 2.3 TX packets
833
834 This part is not going to discuss the whole protocol stacks in linux
835 kernel. We intend to focus on how the driver layer works in the
836 process of transmitting packets when it crosses the POSTROUTING.
837
838 Each NIC has its own buffer for packets (ring buffer). Kernel will
839 write packets into the buffer and send TX instructions to control
840 register. The NIC takes packets from the buffer and hits the wire. The
841 linux kernel will copy the packets to kernel space by invoking the
842 memcpy_fromiovec() function which is invoked by packet_snd() function,
843 which invoked is by the packet_sendmsg() function, which the source
844 code can be found in src/net/packet/af_packet.c, when upper-level
845 protocols have already prepared the packet:
846
847 err = memcpy_fromiovec(skb_put(skb, len), msg->msg_iov, len);
848
849 The initialization of the structure proto_ops is in
850 src/net/packet/af_packet.c. This structure includes function pointer
851 to kernel implementations:
852
853 static const struct proto_ops packet_ops = {
854 .family = PF_PACKET,
855 .owner = THIS_MODULE,
856 .release = packet_release,
857 .bind = packet_bind,
858 .connect = sock_no_connect,
859 .socketpair = sock_no_socketpair,
860 .accept = sock_no_accept,
861 .getname = packet_getname,
862 .poll = packet_poll,
863 .ioctl = packet_ioctl,
864 .listen = sock_no_listen,
865 .shutdown = sock_no_shutdown,
866 .setsockopt = packet_setsockopt,
867 .getsockopt = packet_getsockopt,
868 .sendmsg = packet_sendmsg,
869 .recvmsg = packet_recvmsg,
870 .mmap = packet_mmap,
871 .sendpage = sock_no_sendpage,
872 };
873
874 Linearize the buffer and do the checksum by invoking dev_queue_xmit(),
875 which can be found in src/net/core/dev.c:
876
877 /* GSO will handle the following emulations directly. */
878 if (netif_needs_gso(dev, skb))
879 goto gso;
880
881 if (skb_has_frags(skb) &&
882 !(dev->features & NETIF_F_FRAGLIST) &&
883 __skb_linearize(skb))
884 goto out_kfree_skb;
885
886 /* Fragmented skb is linearized if device does not support SG,
887 * or if at least one of fragments is in highmem and device
888 * does not support DMA from it.
889 */
890 if (skb_shinfo(skb)->nr_frags &&
891 (!(dev->features & NETIF_F_SG) || illegal_highdma(dev, skb)) &&
892 __skb_linearize(skb))
893 goto out_kfree_skb;
894
895 /* If packet is not checksummed and device does not support
896 * checksumming for this protocol, complete checksumming here.
897 */
898 if (skb->ip_summed == CHECKSUM_PARTIAL) {
899 skb_set_transport_header(skb, skb->csum_start -
900 skb_headroom(skb));
901 if (!dev_can_checksum(dev, skb) && skb_checksum_help(skb))
902 goto out_kfree_skb;
903 }
904
905 And queue a buffer for transmission to a network device by invoking
906 the __dev_xmit_skb() function, as you can see below code snippet:
907
908 /*
909 * This is a work-conserving queue; there are no old skbs
910 * waiting to be sent out; and the qdisc is not running -
911 * xmit the skb directly.
912 */
913 __qdisc_update_bstats(q, skb->len);
914 if (sch_direct_xmit(skb, q, dev, txq, root_lock))
915 __qdisc_run(q);
916
917 Finnaly, raise a softirq of TX by invoking the __netif_schedule()
918 function in __qdisc_run():
919
920 while (qdisc_restart(q)) {
921 /*
922 * Postpone processing if
923 * 1. another process needs the CPU;
924 * 2. we've been doing it for too long.
925 */
926 if (need_resched() || jiffies != start_time) {
927 __netif_schedule(q);
928 break;
929 }
930 }
931
932 In the func __netif_reschedule(), the softirq has been raised:
933
934 raise_softirq_irqoff(NET_TX_SOFTIRQ);
935
936 The softirq handler is registered, while kernel is booting, which can
937 be found in the func net_dev_init() in src/net/core/dev.c:
938
939 open_softirq(NET_TX_SOFTIRQ, net_tx_action);
940
941 net_tx_action() calls qdisc_restart() which has:
942
943 HARD_TX_LOCK(dev, txq, smp_processor_id());
944 if (!netif_tx_queue_stopped(txq) && !netif_tx_queue_frozen(txq))
945 ret = dev_hard_start_xmit(skb, dev, txq);
946
947 HARD_TX_UNLOCK(dev, txq);
948
949 dev_hard_start_xmit() will call the func rtl8169_start_xmit() by
950 invoking *->ndo_start_xmit():
951
952 rc = ops->ndo_start_xmit(skb, dev);
953
954 Then your dirver's implementation of transmission
955 function(rtl8169_start_xmit()) will be invoked. The process of the
956 figure is displayed below:
957
958
959 Figure 4: The process of TX packets
960
961
962 $=======================$
963 -----------------------$ Upper-level protocols $-------------------------------------------
964 $=======================$
965 |
966 |
967 \|/
968 +------------------+
969 | packet_sendmsg() |
970 +------------------+
971 |
972 \|/
973 +--------------+
974 | packet_snd() |
975 +--------------+
976 |
977 \|/
978 +--------------------+ +------------------+ No queue
979 | memcpy_fromiovec() |----->| dev_queue_xmit() |----------------+
980 +--------------------+ +------------------+ |
981 | |
982 | Y |
983 \|/ |
984 +------------------+ |
985 | __dev_xmit_skb() | |
986 +------------------+ |
987 | |
988 \|/ |
989 +-----------------+ +---------------+ |
990 +--------> | qdisc_restart() |<---------| __qdisc_run() | |
991 | +-----------------+ +---------------+ |
992 | | |
993 | | |
994 | \|/ |
995 | +--------------------+ |
996 | | __netif_schedule() | |
997 | +--------------------+ |
998 | | \|/
999 | | +-----------------+ +-----------------------+
1000 | +---------->| net_tx_action() |---->| dev_hard_start_xmit() |
1001 | Raise a softirq +-----------------+ +-----------------------+
1002 | | |
1003 | | \|/
1004 | | +----------------------+
1005 | If Queue is not empty | | rtl8169_start_xmit() |
1006 +--------------------------------------+ +----------------------+
1007
1008
1009
1010 ----[ 2.4 Three ways of packet's traveling
1011
1012 Linux kernel supports many network protocols which have different
1013 implementations. We will only use IPv4 to descrbe 3 ways of packet
1014 flows which bypass the netfilter.
1015
1016
1017 ------[ 2.4.1 Network to Host
1018
1019 Firstly, register the handlers for different protocols by using the
1020 dev_add_pack() function while kernel is booting, such as IPv4's
1021 handlers registration in the inet_init() function which can be found
1022 in src/net/ipv4/af_inet.c:
1023
1024 dev_add_pack(&ip_packet_type);
1025
1026 Which the structure has defined in the same source file:
1027
1028 /*
1029 * IP protocol layer initialiser
1030 */
1031
1032 static struct packet_type ip_packet_type __read_mostly = {
1033 .type = cpu_to_be16(ETH_P_IP),
1034 .func = ip_rcv,
1035 .gso_send_check = inet_gso_send_check,
1036 .gso_segment = inet_gso_segment,
1037 .gro_receive = inet_gro_receive,
1038 .gro_complete = inet_gro_complete,
1039 };
1040
1041 After registering the protocol handlers, the .func function pointer
1042 will be invoked by netif_receive_skb() in driver's softirq handler (we
1043 use rtl8169_rx_interrupt() in case):
1044
1045 if (pt_prev) {
1046 ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
1047
1048 After packet's sanity checking, the packet goes to the
1049 NF_IP_PRE_ROUTING hook for filtering rules. Then it will enter into
1050 the ip_rcv_finish() function, which looks up the route depending on
1051 destination IP address. If the destination IP address is matches with
1052 local NIC's IP address, the dst_input() function will brings the packets
1053 into the ip_local_deliver(), which will defrag the packet and pass it
1054 to the NF_IP_LOCAL_IN hook:
1055
1056 /* Input packet from network to transport. */
1057 static inline int dst_input(struct sk_buff *skb)
1058 {
1059 return skb_dst(skb)->input(skb);
1060 }
1061
1062 In the end, invoke the protocol handler in the
1063 ip_local_deliver_finish() function:
1064
1065 ret = ipprot->handler(skb);
1066
1067 Then, the upper-level protocol will continue to process the packet.
1068
1069
1070 ------[ 2.4.2 Network to Network
1071
1072 After the filtering of the NF_IP_PRE_ROUTING hook, look up the route
1073 by invoking the ip_rcv_finish() function, and through the
1074 skb_dst(skb)->input() enter into the ip_forward() function which does
1075 validate checks including checking the packet type:
1076
1077 if (skb->pkt_type != PACKET_HOST)
1078 goto drop;
1079
1080 Decrease the TTL, check whether the packet is allowed to defragment,
1081 check the length of the packet which should not be bigger than MTU,
1082 etc:
1083
1084 /*
1085 * According to the RFC, we must first decrease the TTL field. If
1086 * that reaches zero, we must reply an ICMP control message telling
1087 * that the packet's lifetime expired.
1088 */
1089 if (ip_hdr(skb)->ttl <= 1)
1090 goto too_many_hops;
1091
1092 if (!xfrm4_route_forward(skb))
1093 goto drop;
1094
1095 rt = skb_rtable(skb);
1096
1097 if (opt->is_strictroute && rt->rt_dst != rt->rt_gateway)
1098 goto sr_failed;
1099
1100 if (unlikely(skb->len > dst_mtu(&rt->u.dst) && !skb_is_gso(skb) &&
1101 (ip_hdr(skb)->frag_off & htons(IP_DF))) && !skb->local_df) {
1102 IP_INC_STATS(dev_net(rt->u.dst.dev), IPSTATS_MIB_FRAGFAILS);
1103 icmp_send(skb, ICMP_DEST_UNREACH, ICMP_FRAG_NEEDED,
1104 htonl(dst_mtu(&rt->u.dst)));
1105 goto drop;
1106 }
1107
1108 Then, by invoking the ip_forward_finish() call the skb_dst(skb)->ouput
1109 entering into the ip_output() after the NF_IP_FORWARD hook. Finally,
1110 it will arrive the NF_IP_POST_ROUTING hook.
1111
1112
1113 ------[ 2.4.3 Host to Network
1114
1115 This is the last type of the direction of packet's traveling. When
1116 userspace program uses socket to send the packet, the packet will
1117 traverse a lot of functions into the NIC driver in the end.
1118
1119 The ip_queue_xmit() function is the key function in network layer
1120 which is provided by linux kernel. The packet looks up the route in the
1121 ip_queue_xmit() and sets some segments, such as defragment flag,
1122 then passes it to the NF_IP_LOCAL_OUT hook for filtering:
1123
1124 /* OK, we know where to send it, allocate and build IP header. */
1125 skb_push(skb, sizeof(struct iphdr) + (opt ? opt->optlen : 0));
1126 skb_reset_network_header(skb);
1127 iph = ip_hdr(skb);
1128 *((__be16 *)iph) = htons((4 << 12) | (5 << 8) | (inet->tos & 0xff));
1129 if (ip_dont_fragment(sk, &rt->u.dst) && !ipfragok)
1130 iph->frag_off = htons(IP_DF);
1131 else
1132 iph->frag_off = 0;
1133 iph->ttl = ip_select_ttl(inet, &rt->u.dst);
1134 iph->protocol = sk->sk_protocol;
1135 iph->saddr = rt->rt_src;
1136 iph->daddr = rt->rt_dst;
1137
1138 Then, invoke the ip_output() function by dst_output() function which
1139 can be found in src/include/net/dst.h:
1140
1141 /* Output packet to network from transport. */
1142 static inline int dst_output(struct sk_buff *skb)
1143 {
1144 return skb_dst(skb)->output(skb);
1145 }
1146
1147 After filtering in the NF_IP_POST_ROUTING hook, the packet will
1148 deliver to the lower layer handlers.
1149
1150
1151 ----[ 2.5 A bigger picture
1152
1153 Now, hope the last figure can help you understand the complete process
1154 of a packet's rx/tx:
1155
1156
1157 +-------------------------------------------------------------------------------------------------------------+
1158 | A P P L I C A T I O N L A Y E R |
1159 +-------------------------------------------------------------------------------------------------------------+
1160 /*\ \|/
1161 | +-----------------+
1162 | | ip_queue_xmit() |
1163 | +-----------------+
1164 | |
1165 ipprot->handler(skb) | \|/
1166 +---------------------------+ +----------------+
1167 | ip_local_deliver_finish() | | ip_local_out() |
1168 +---------------------------+ +----------------+
1169 /*\ +---------------------+ |
1170 | | ip_forward_finish() |------+ \|/
1171 $================$ +---------------------+ | $=================$
1172 $ NF_IP_LOCAL_IN $ +-------------+ /*\ | $ NF_IP_LOCAL_OUT $
1173 $================$<---| ip_defrag() | | | $=================$
1174 /*\ +-------------+ | | |
1175 | /*\ $===============$ | \|/
1176 +--------------------+ | $ NF_IP_FORWARD $ | +--------------+
1177 | ip_local_deliver() |----+ $===============$ | | dst_output() |
1178 +--------------------+ /*\ | +--------------+
1179 /*\ | | |
1180 dst_input(skb) | +--------------+ | \|/
1181 +-----------------+ | ip_forward() | +---------> +-------------+
1182 | ip_rcv_finish() |-------+ +--------------+ | ip_output() |
1183 +-----------------+ | /*\ +-------------+
1184 /*\ \|/ | |
1185 | +------------------+ \|/
1186 $===================$ | ip_route_input() | $====================$
1187 $ NF_IP_PRE_ROUTING $ +------------------+ +---------------$ NF_IP_POST_ROUTING $
1188 $===================$ | $====================$
1189 /*\ \|/
1190 | +--------------------+
1191 +-----------+ +----------+ | ip_finish_output() |
1192 | arp_rcv() | | ip_rcv() | +--------------------+
1193 +-----------+ +----------+ |
1194 /*\ /*\ \|/
1195 | | +---------------------+
1196 +----+ +----+ | ip_finish_output2() |
1197 | | +---------------------+
1198 | | L A Y E R III |
1199 | | |
1200 --------|-------------|-------------------------------------------------|-----------------------------------------------
1201 | | |
1202 +-----------------+ +------------------+
1203 | net_rx_action() | | dev_queue_xmit() |
1204 +-----------------+ +------------------+
1205 /|\ |
1206 | \|/
1207 | +------------------+
1208 +-----------------+ | __dev_xmit_skb() |
1209 | __napi_schedule | +------------------+
1210 +-----------------+ |
1211 /|\ \|/
1212 | +---------------+ +--------------------+
1213 +---------------------+ | __qdisc_run() |------>| __netif_schedule() |
1214 | netif_receive_skb() | +---------------+ +--------------------+
1215 +---------------------+ |
1216 /|\ \|/
1217 | Got a packet(s) +-----------------------+
1218 +----------------------+ +------------| dev_hard_start_xmit() |
1219 | rtl8169__interrupt() | | +-----------------------+
1220 +----------------------+ \|/
1221 /|\ +----------------------+
1222 | | rtl8169_start_xmit() |
1223 | L A Y E R II +----------------------+
1224 | |
1225 --------------|------------------------------------------------------|----------------------------------------
1226 | |
1227 | |
1228 | \|/
1229 $------------------------------------------------------------------------------------------------------------$
1230 | H A R D W A R E L E V E L |
1231 $------------------------------------------------------------------------------------------------------------$
1232
1233
1234
1235
1236 --[ 3. Conclusion
1237
1238 Netfilter is an excellent framework on both design and implementation
1239 for networking. But netfilter has one well-known flaw is that the
1240 connection tracking costs too much. I've tested a machine with two
1241 4-cores CPU, 3 Gigabytes memory, Intel 8xxxx NICs. The result of the
1242 testing was that processing the bidirectional 64-byte packets was
1243 640Mbps without connection tracking and 330Mbps with connection
1244 tracking. It seems to need optimization if your requirements are
1245 performance-sensitive.
1246
1247 Nowadays, many commercial companies are using a cheap combination of
1248 X86 hardware and GNU/Linux to develop their networking products. But
1249 there are still many commercial networking products violating the
1250 GPL. That's shame!
1251
1252
1253 ----[ 3.1 Gratitude
1254
1255 This article is dedicated to my neurons. I can't do nothing without
1256 their faithful support. And, I must thank those great hackers
1257 including Phrack's authors, RMS, John Carmack, LulzSec, etc. You guys
1258 really inspired me to keep hacking on Purpose/Hack/Life. Finally, I
1259 thank my beautiful wife for proofreading the article and helping me
1260 fix the grammar errors.
1261
1262
1263 May L0rd's hacking spirit guide us!!!
1264
1265
1266 --[ 4 - References
1267
1268 [1] Hacking the Linux Kernel Network Stack
1269 bioforge. Phrack Vol 0x0b, Issue 0x3d, Phile #0x0d of 0x0f
1270 http://www.phrack.org/issues.html?issue=61&id=13#article
1271
1272 [2] GNU/Linux kernel 2.6.33.4 source code
1273
1274 [3] Understanding the Kernel Network Layer
1275 Breno Leitao& Arnaldo Carvalho
1276 http://stoa.usp.br/leitao/files/-1/3689/network.pdf
1277
1278 [4] Netfilter/Iptables Firewall
1279 http://www.netfilter.org/
1280
1281 [5] Kernel Intrusion Detection System
1282 http://sourceforge.net/projects/ids-kids/
1283
1284 [6] Sreekrishnan Venkateswaran
1285 "Essential Linux Device Driver", Chapter 15: Network Interface Cards
1286
1287 [7] Kernel Documentation
1288 /usr/src/linux-2.6.33.4/Documentation/driver-model/*.txt
1289 /usr/src/linux-2.6.33.4/Documentation/PCI/pci.txt
1290
1291 [8] Handling Interrupt Descriptor Table for fun and profit
1292 kad. Phrack Vol 0x0b, Issue 0x3b, Phile #0x04 of 0x12
1293 http://www.phrack.org/issues.html?issue=59&id=4#article
1294
1295
1296 begin 644 netfilter_hacks.tar.gz
1297 M'XL(`,($"4X``^P\:W/;.)+Y*E7E/R#>22+:LBS)KSDK]J[CR(EJ'%MK.YO:
1298 MFZ18-`5)7%$DEZ#L:&=\O_VZ&R`)/B0GMTYV[TY,8DM$H]%`/]#=:,3CT=!Q
1299 M(QZ:8\N>B*TGW^%IPK._NTN_X<G_IL^MUEZSW8:_^]M/FJWV?FOG"=O]'L3D
1300 MGYF(K)"Q)Z'O1\O@'FK_7_IX.?Y[0^$YP^&CRL'7\K^UO[N_B^];.]N[*_[_
1301 MD&<1_]];$PX-_#'&>(C_.SL[Q/_F;GM[>[L-_`=9:#]AS<<8_*'G_SG_?^E>
1302 MGG?/S#>]R\.MF0BW1&AON8XW^[+9;NPUMK<;.U4%\JY[_*9[><4.V:;[4RWM
1303 M9VPYGNW.!KQ:]6_^MCD].%0RU/"K_8]O`/ZGFAASUV7!W<"H5BW7/:A6?JJ]
1304 M/_ZE:[#-$Y9!QMX?_E2#;@:;^H.9RT6U,K)MMNFS$8^"._FS85>KMLLM#Q"%
1305 M4[89#MEZ8^)K7_3/@*BQGGQ_3VBU%VJ<^,U0#I$TP_M_-8^^Y[-(_V,>VH\P
1306 MQG+];S=W]YJ)_K>D_K=V]E;Z_R.>K75V/78$"T)_%%I3=N>`HHJQ?\?F_HQ9
1307 MS)D&+I]R+[(BQ_>8/V3#*&`D&SQD=V/''K,;2_!!E:VS1)C8$'#Q.S^<--@[
1308 M*PCF#*7+\4;/``PAI[XWX?--VQ_P\.!J;-UY+!IS=MFT)W^R_6#N\F%49Z]L
1309 M)YH'=W\:32W';=C^]`C[XC^BV0^=D>-9+D,TT-^*F!.]%#"X/X5A:4!">X8&
1310 MC?W"0X^[[)Q'2!BAN8H`2O:\"YTHXAZ[F;,;QQ_ZX8@WKL>6-Q')]SJS;OQ9
1311 MA"L3LE'(H9<51@Y8HH::5@^MB3-T^(#&%0!H<TG>%#94((\-X9\C9SLA@LC0
1312 MLF,/NG`104]+$&UW:#(!\NWY!VF1$:Z]6V?T^S\:>\R"/K&55A1L5:M_4.:8
1313 MO:)>6]*^-<9'A18Y?EF+F-S,0/E+6ARO]&U0]C:R2U\[]K3T/4C/@-\Z=BFM
1314 MB6C)QN&`#YEIJHW#K/YAYN7?+$)@.L'M#F'A'O`*%@QZ.AY?UMD9FE98/IFA
1315 MR8&7X8*V`.2+1]A8?7_QYL-9USSKG73/K[JUM;?]LS6CDP[__OAM[\0\N7C3
1316 M98PUO^R^3EHNN_VSOYI7O?_$ENV]M$OOY'W?[!__]>SB^(UJKXTCWQ,U)S#'
1317 M@[`F;HS-H\B/3)=[!OM4K50J3#Z;3#C_X/ZP)J)P9H-(!@#_``CP#8%@#T>K
1318 M\:Y[U66WECOC`M2`LYE`J?=!JGE`XOWAJGOI@1D@.>T?7UV!V@W8S(L<%^4;
1319 M0.;4\>\S'H+&--B%Y\Z9[\F>6]B#!9832IMTPQDX$:`=$8(0`D?ACMO1)0B!
1320 M"-^S-:R@$P*ME\WL,5CZ=:`S)+(.V?F'L[-.MC6PA"`Z<ZV.%S$P8K?<))(.
1321 M6;-38;`,[ZT0+(0S!'5EE@LV83`G,#"=!(@:B:L5*G/D>&!*IF1,U81=?T03
1322 M3I8)[,5T"E^$-$PC'U<55@OG+)#P7I]9@T'(A:`^UR=]%OAAI$]UY@EGY,%:
1323 M(-VPP8%/`W)/9!=`P-J'"1`BDF!(]@?%TP$7=NC`$J/E.T^L_-CW8?(T*DF(
1324 M-S3QE>D'@L&.PNE;)Y8GP-?S8&ZX#(OZ^"(J=+J812-?=2*J3F>>3;L1K8]+
1325 M1*`M]IB8F&BXU%X@V,3S<6?Q43J@^?0:5DIJ)"[F&?4$?B3BFO(`;+@[D.N+
1326 MEELP,0NYE-D;/QHCCSAM-$J426['0#Z:;P<,BVU%M/S4J!;WRY<O,>*45;>^
1327 M,P#AX_;$A*TU5K9X)NM@BHWJ;U58"M4"5A6TD*W#[PZ^EG([L"*+OB+#0=LE
1328 M#]5W)_F&_Z`C?*_ET!DU&&GS"/&P#988D`E:$&?LPBQW#(-P$`@@D`,;M1H,
1329 M82!2[`8?:_`9$/E(O>Q4E8P\]^_J2E64BB2ZJ%2GCJOE:0")Z#80!T-]@VU4
1330 M\0)X')'O0BQ%?H.<1@[*=,Q4TA=D"RD'+L>0U>)1#<))KS+3':!VL6>'FN+\
1331 M_CNC6:D=/6U#?5%X&`MY-`N]9*'EE%D4SE$$!><X%%$,?RTI<$"G,G0TAY1(
1332 M8(\'!K>&JUUG:P2\5F>[!CL$9AKL-Y8\4E'E0DH$,9<V#MEN1[U0OS(+D"$Y
1333 MA0&WSN6LMEY3TN`8...7G\*7[,4+5GCMO:Q*=2UK;%(?A[UB+:#]-X`$Z=S8
1334 MZ,`'!W_)$>^+1-8T0SV90M3HVS44[`W6KK.WIWVU6QNT(&BK#8C9,K.9\JG@
1335 M48*G#KMJLUEG"HFA@=G!7`.32XY;9@RRGA*S00U,3DTW4T@"N'`AV#=0?<6(
1336 M>\9=P<L82ES/,Q30].,-J(R3U=22#G7U&5O">QF!E8.IP7XR`@V8\T@JT\#'
1337 M)A1";`$[&DOK.NU1^=%T^4A6-B<F"!/W_'>2(5UZM(W\GY*>&,\#TI."E4E/
1338 M0LQC2<^?/_2N07AV$N%))>//,X@PE`L!K+&Y<XMN4*_4[MY`../YB1C44]D@
1339 M9XIL:&JD"2$L"N.W/)R#)9.;<KKJB8-4D(J,6`$SGZ7R@_R<#$/.4[N$G"WX
1340 M:96<CY)S:RHY]PQ>P-\L%??:JL9+ID%("4)2GR7(E7'7!RMN%IU\/VU;R!&=
1341 M[B*=9#1]89)UD;V3.8&<5S1Y>9?ZF,#;D>?#CNAC("*4AUW!9SW>(&%-P$9(
1342 M`X$[+6V'6UL%MLEWZ+S!3CZIK=$PB?0BR#/&/ARPY^"LL3[^_N2!'*:V,R&_
1343 M4[W/.FLV**&,;E&H^A=7U^;EQ8?KWOE;5G,M$1GD3L)D(NG0DTN$&SEZ6.BY
1344 M1:$U'(+#1%9-@'3:8^4$J`;<],L]Z86N\9T5V6,3@OI:YC52XLVF]30:*G'+
1345 MM%8;8JXHAH%(TY2!+%MWO*^!@O$U,"2@MNY/AIY1\`:-K"^8DG,#L@4T=98X
1346 MBLHET3VHV!6A$$)Z($,G%+H?HD>20>A'ONV[:)5[_?[EQ?6%"5V5J$I%8N>G
1347 MYO')2;=_G9'7<S\`Z?#\B,:2Z)<YHS"@=$8-W1LM.J.:GR4%)O6T*!>4\?JU
1348 M:4D?%=Q%G(N,F=LMX]MF@HC3F=#>'0H*$O0Q42I#[O);"SWQ-/Q3GK#<GI7_
1349 MJYS<@G?L#;)*J'NSSS3ME6*6QA(WF47ZR"G>'CJ>(\8<X^9H3-1*2G'3P@P:
1350 M1IQ(7B1`/>9JI,*2H'JC?O=#W^:#Q"$GA>(8`JE(#W,4&0W%`=?>6R/'7M.#
1351 ML8^HU7'<%F];=5RAZ(Y;$^G-3VZ4<--XR&E8&0O``W>.2&XL*0$('/*_SR@<
1352 M&*FXC#)LKJO%KY)BP!OY+H]#;$73<HOA>`\8C*7VXJO,Q==9B\<R%BJY`R3`
1353 M!RVNM`.30DM=`X!5L/QWF-@,YM(]A35W/,D')3"9Y<%=$+=)#<DUG\*6:(%;
1354 MVNO#\KD#+I,ELO6-C_C!S?#RRK"5400<&$;%O14<WC\N4XNB!--K-6!/2',(
1355 M]H(D5DK!'[_!$&*W94,1'EA<S=PEBVXDY@Z]WA)CE\300*=#CKW,%Y;1"4@W
1356 MCRCG#-1I646(865;-`^HC=*'W9-W%Y)J:"\F%%]IV4<CYS/E)GBOF9J+B07!
1357 M_%09`\P:D562V1:I_1X78@T-Z1VR>N@,!J"#:)*4$90*K_(Y*%X@`'+F:?ZK
1358 M3EI=+AIUA4?<60%V4:$[=I!I`FF%%28N<SV4RP5E2U\K++"R'A@E(5!BR920
1359 MO0HM3TP50V)[CZ</I"&QQ5$H4KN3F!P'[$Q_S.\:C=BBDYYH/B8*@4A=S/S;
1360 M'.2@%'*@(*4.DB2AN`63R"11.`1GZ>27[K4)[MC;"W#(%`PP`UPL*9K\5LI-
1361 M+`.V!3O=\67_W>4;L]_O'V1W2,E.-+T>YP,!FUPF5+B!S6[2*>`YN[CHOP9*
1362 M#@HMW>MWW<N#:N4W"A(3TT(F*C+'=SBQ7P'(/#[KGG_N$!3]0'_'OY5[,JE7
1363 M`)20Z??EQWBW`&(GS+7FF-6$G5@:HXI:J3CCE1EXW0!AB1>YD\!2?'C(D)AW
1364 M0$R';6W%:71HG5IV(Y8QU4D%D/$TZJRFX=T\&ILHK4:=Q=/+=EL`G'\MA=_`
1365 M?8-])::X"TLITSJJ]56<K,C`"E[K_ACN#GE]);7YD(0+]#516^"'GW75U/:3
1366 M23>J#V1+-Q:=49!TJ9DI)'7V(HGBZF1366D^,-L+AMC1`YS67MHQ%ZL5.K:;
1367 M:4"4]E1+=*V\[SNR%11,"QGY#!R5*50')#+['?EQ#O04OF*@9X<^6*A!:CQ`
1368 M2TUP>F;<_`)&2<JEQA",L556%'RK!>YE&N2GX;3$7HC1DW=:T(?JF0;NA=,4
1369 MEC]$2>*`9:&]C$KCF#3)D,$N%#K@(PP@!DV7-MV;KJXO4`/14Z4LN.=$ICP-
1370 MK4G/*,:(+8[E.O^0SJ(_$0G&^!2C@3^(RX>)"Y@%"(;*`H(Y/35[Y]WK7'OH
1371 M^*$3S7$MH!WVV<N>>=J[O+HN#@2N)-/@NG&DG$PR.2@IT@7N82<+LHBR%.`!
1372 MTC*#96G3POB$.&]HAGSD"*HJ@1ZU%_'DY)J6M,<C%-C8)/;),Q*L^)D%Y2R\
1373 MF?,<YV"4F;=\G%*0#*D+-+U,[$LL24%C[O_M:HD6U?^H.JM'&>.!^K_]O1VL
1374 M_]G!\M_]O=V=)\W6[MY>:U7_\R,>V!J0U2#)#9M._+'>A\TBQT5[$%'9W:+8
1375 M"SW=I_'Y9^K=8A`^\,'/QR2(JA1JL+]P<)PG[FPP@L``T\X<WML11/D-PO$>
1376 M5!."1]C'G2#`$SQTHZ?SV&W'N*'G#?V^,R)XZO.Q6+.#90MX*`^.:[O9W(:M
1377 MZVGUJ5:9(>9B"_U8T1@?9=Y'`\<OOG.=F]S+F0?68E"`!#,TRKWD8>@54,+P
1378 MPE<U(7H#%K_DA[+"P-IR/`6++4-5Y?+AJFN^OGH#KUA2O*+>90)U"*@PZT31
1379 M2>@'B7,K:%UD`0PBT<G``65!3XX^U6"J^AVD2,_O4DQ,2TO(RZL,9!0XF^)!
1380 M,8.=!(4)O,(<U#H&?48'1T"XJ>5XU`&\`KNN$A/P^?;7S\;3ZF]/R=AF@X(!
1381 M%K/]VM[=^ZSG'/K@G(-4HRQ9K(^99W34J.X-22ZB";E]B[1(3!(B*9AAZTZ`
1382 MRZD']/#2H+%SP.BK4FHEWX$:C!KU*7JS@6%D$:'LH"\-W@<3H;VXD4*]+`V>
1383 M27'@=&XN;@1%O\TT`TI8-#/"T:C!1!(Q?Y2A5!LZIIA<+IB>B6WDQZ7OL5CB
1384 MD+6T-^M!%)KR]0OXI5I4.^RLR'KVBFU#_/FT6AG2MH^C@S"'>#PNK!$_8'@N
1385 M$?*I'W&(.:;S7I].)TA4FI^1L`K_`HYQ*Z;Q'D5,"<=;C/J95,Y$'.@H49^%
1386 M;*\I/ZK.KBY.?C$OCS_6LWD@`VAM+J#UQ)^Y`TSZ@D)Z++3N%-)GZC0%^E1@
1387 M60'8#VMD1(P'2!?RK)>]>W,)9)V<`699K>FI,L3<G&K0`5\"6#HY;09]_&P"
1388 M-D1&]("C)?E3CQD/G_59VJXO>(J,"%X\<R18H5>T:I-7^LJ^:@U2*4'Q:P@0
1389 MY*$U=5ST98^5LYMKEQ_,.'V"J57\7",Q:7TV,K*GM&4A?#L/+T^-29_C(V,P
1390 M'C&0:H[M2@E$PM131]9_J@A:U0W%QS.*E61/-H\PV^.B5[[;*33<DN._4VR(
1391 M?*&K9=H@RX=^+ND"7G@F#T+FV@@A9JP916BL`"H=((I<5/_VS\6F@*C5=:D(
1392 M`_M'.5ZP4="0-7!:\T!@;)F1@_RB4YBL+SSF@I.2+851F7'`B>*N4FA),K53
1393 M"D7YV$.JZUP`,)'32G;(G78]O](JZX&]2L3%LUP7?"LZ$]'RD3'9FC;B"0);
1394 MN\+M'P)>=5#2:#0H=DH-'V**?-U"*+'^>0<$M\[RQA_(>T&)HE2)RW>)I>;Q
1395 MDW=J.7A2+++T/60>DX`Q9Q\+\_YH.70J)`_F`G=>G#GJ)SJ@^MQ3G05M73A_
1396 M$$%I,%]D-LSR"4MZU'3!T8[D=/$013+OH3F76=V"B:SHBI!W5N)9Y82RQ$6)
1397 M(9=Y*2H#]B)Q(V"1]'P=H=Y@/QMU"5])1"0=3BJFD17PHO!B_MPC'M*%!!@0
1398 MO-OGXH!V?#+27N1;M822!&$!59R-/,#6["Y4FG14DVBU%Z.,4U3?B++]<V[:
1399 M)?Q-&S6!OU?.^(DR'ILC[O%0'G`,E8-.E146Q%96J"J+T0-^R;'`U1[#B@EY
1400 M%DV1E11""/\01LB38L>[M5S8`6H.;S`^#:*Y(>UC/*HRE*!.A.,48T3N^;/1
1401 MF/78"'28ZC`>(S2(O?ZDT?5!<Z"S7"#Z`:)1BW>*&!T[TKYL;AIR(;$CYNP)
1402 M]\9&1QI4$!KF\2\1D]E;RE1#.!32;9CD'$%95_HA1Z-!CXXPW8NE"_CM!=C]
1403 M4W@4_]1P&J!&MN+J?\G)W'_O^#^?_XGO2`Q-)WBL2\#?=/^[A?=_=[>W]U;W
1404 M?W_$LY3_CW0)^,'[O\WM^/Y?N[V#^;_])HC!*O_W`Y[O<?\WD:%'N@'\W>[Z
1405 M_M^_WOO@LU3_4T;^4P<!#^7_=_?W$_O?VB?]WVVN\O\_Y%E^_Q=_@Q,X$[SL
1406 M;B\VR7=TJV!=BSH%YN0AP$C*:.J(@#J1WS2T;%5Q$[^.2ZK`1ZQ2":1V:[C.
1407 M3L%Z^'="71\NW"!FJRO$CW.%.%T9&A$+XESAL[?],P@4GGWK!>/D6.0;KA[3
1408 MR4.E@AD-//+I:^4XY?>+$U@L+%X,O/A*\[_1O>,'+PF_[N+9_/FUV>O7**K%
1409 M&[NU?&$296&,7YN?Z\N:6\N;V\N;MS]K5/6NKUI[YF7W+^`@=!5AM1I]8*]>
1410 M0;S/?F?R&\0[$.2J*\2J`H;D/"GKQ>*X,;@(9'6HO%.>SU,581S,:M6Y)?=(
1411 MO>'8EQ=8>6-T4+0\;(U'XR;F"1(+Q=9:[?U&$_ZTUF1!3%SBF;\_/`C]`+9$
1412 M\&HD%CF0=B?W+CYU\QG")A81;0JF)F)Z-F_F$0<+!`);>^V,-E$L+,\H*SS6
1413 M1\::F+5/7_:'G[XTF^I?"ZE0"P&&$Z_44!=,:_RZW<:*N#PN[LVIVJ93O<_V
1414 M%6(LIY28=C1HQ35<4!^MA/UF#FNTN$:ZLK!`NE)6^.S<:M71I1#9TNB*7A?-
1415 MR@NC*V7UUT,\.\,D/RQ!A0Y_``KO75'+YI&Z("A%P$AO7U4J<2$*MN'YM<J^
1416 M@J@^%S+!F/8#=E32,J4WEQ=]+*&+KR7I;7%Y;>6^JN[39+BBR>_#[`CHU&()
1417 M2]2ARP*V4#[J0=8L@,JSAW*-7\VBDLKU2I)\E%7KP9@X%HPS%;'X+F:,R[T1
1418 M9K$$>SZ02<)@3`76R`Y9X#U.JFH/\<JGOE+KAN)>4&1WRFUR&[`>[_F@$?_%
1419 ML:K)J7RLQ'6\Y:CT>8.U#!U$:V@;6<!MXW\F/"6R0W=^BD8.)RMOD,#VJO[_
1420 MBN5W'99==?B:FPY?=='AZX4EN?:$OD"YH%1R2<69-(1+Y:TH6I4HERE/;B\Q
1421 MO9R_%LM9<AN^(DLSY,T?VBHL;RYO@*JTHHC].S#6#OT'!HDC1+6>)*_/@(IR
1422 M?E,K#%O>3.._7G8'+!Z!*%]TZ6N)I9*6,Y*G6U21G-.H^#0+%DJ,&\E>9$BD
1423 M,U4%FG,K='R=1`65WB2:B!?*-&T$9>SU#S1U9'U`?O!\QHZ#P,7_I0&\B0.9
1424 MHH]54'>R4J,`FDB$U7&';,375$M4L5JR*O=57;^H$C4)KJF2L5):BXHB0%6-
1425 MSV198[6B#I!?``WQX7%\&B/&9!QPPPKFM9C(.EN#CVO4HJ^U]"+0>VCMK2FA
1426 MP-SV]";Y?YVHM!3#0!%P6X8D*!GH7<FRT\."I>AHS1ZEPTMJ6"5(,-3J47%T
1427 M](_3,#`=:EEU*H+DBTFQ$ZU5IH0T\[]MJ$K2!2S(U9)6RLI$U2"`6(8_)O*M
1428 MEF4LM*M&.I,KC`KM_^KP?_6LGM6S>E;/ZED]JV?UK)[5LWI6S^I9/:MG]:R>
1429 AU;-Z5L]_MP<')`````""_K]N1Z`"``#,!=C(SDX`>```
1430 `
1431 end
Attached Files
To refer to attachments on a page, use attachment:filename, as shown below in the list of files. Do NOT use the URL of the [get] link, since this is subject to change and can break easily.You are not allowed to attach a file to this page.