Netgear® DM111P hacking
This document describes the process of building a Debian-based custom firmware image for the DM111P ADSL2+ Ethernet modem. The original intent was simply to adapt the original firmware to provide IPv6 connectivity -- however digging a bit into the problem revealed a large optimisation window, be it by removing services that I deemed unnecessary, or by adding other, more useful services (such as SNMP).
We will adopt an incremental approach here : first we'll describe the device in some details, then we'll cover the compilation and booting of a custom kernel, then we'll examine a possible root filesystem layout, and finally we'll step through the creation of a full firmware image, which will be flashed. Needless to say, the whole setup will be based on GNU/Linux :-).
Description of the Netgear® DM111P
General description
The DM111P acts as an ADSL2+ modem by bridging the (ATM-based) ADSL connection to its integrated Ethernet 10/100 port there are several ways to do this, which you can configure using the integrated web interface basically, either you let the "modem" handle the PPP connection (PPPoA Bridging or PPPoE Bridging), or you handle the connection from the router to which the modem is attached (RFC 2684 Bridging). Which one you choose is a t your discretion, but will generally depend on your ISP, the router you have, and the functionality you want (e.g., if the DM111P itself does not support IPv6, the you'll have to negotiate your PPP session from the router, and hence use RFC 2684 bridging). Note that if you have the choice, PPPoA encapsulation is generally the way to go since it allows for a full Ethernet MTU.
The board is based on a Broadcom BCM6338 chipset, which combines an ADSL2+ transceiver, an Ethernet controller, and a MIPS32 CPU. The main components on this board are shown on the schema below.
We'll be particularly interested in the serial connector, which allows one to have access to Broadcom's Common Firmware Environment (CFE) -- this is the topic of the next section. Documentation about the BCM6338 also mentions JTAG capabilities, but I haven't seen any connectors which could provide JTAG access, so just forget it for now.
Accessing the serial console
The serial interface is a 3.3V TTL, left-to-right pin assignment being Vcc, Ground, Tx, and Rx. To be able to connect to it from a standard PC serial interface, you'll need a 3.3V serial to RS-232 level converter, which can be purchased, assembled from a kit, or built from scratch.
If you wish to build the level converter yourself, you'll need a MAX3232 and five 0,1µF capacitors -- for reference, a schema is shown below.
Once you're done, you should be able to get a console connection : fire up minicom -- the serial port setup is 115200 bauds, 8 bits data, no parity, 1 stop bit. If everything is OK, you should see the DM111P booting :
CFE version 1.0.37-0.6.8 for BCM96338 (32bit,SP,BE) Build Date: Tue Apr 10 10:08:13 CST 2007 (michaelc) Flash Config: CS0(1fc00008,1f),Base(bfc00000),Size(2MB) Ethernet Network Device: Internal PHY Auto-negotiation timed-out Board IP address : 192.168.0.1 Host IP address : 192.168.0.2 Gateway IP address : Run from flash/host (f/h) : f Default host run file name : Default host flash file name : bcmModelName_fs_kernel Boot delay (1-9 seconds) : 1 Board Id Name : RTA1320 Psi size in KB : 24 Number of MAC Addresses (1-32) : 1 Ethernet MAC Address : 00:14:6c:2b:94:31 Memory size in MB : 8 ==== Press space key to stop auto run (1 seconds) ==== Auto run second count down(before hit space key): Code Address: 0x80010000, Entry Address: 0x80171018 Decompression OK! Entry at 0x80171018 Closing network. Starting program at 0x80171018 Linux version 2.6.8.1 (compiled by michaelc) (gcc version 3.4.2) #1 Fri Jun 1 12:20:59 CST 2007 man/device id ec/2277 FLASH_BASE bfc00000,blk 0 Total Flash size: 2048K with 39 sectors NVRAM @0 block Scratch pad is not used for this flash part. CPU revision is: 00029010 Determined physical RAM map: memory: 007a0000 @ 00000000 (usable) On node 0 totalpages: 1952 DMA zone: 1952 pages, LIFO batch:1 Normal zone: 0 pages, LIFO batch:1 HighMem zone: 0 pages, LIFO batch:1 Built 1 zonelists Kernel command line: root=31:0 ro noinitrd brcm mips: enabling icache and dcache... Primary instruction cache 16kB, physically tagged, 2-way, linesize 16 bytes. Primary data cache 8kB 2-way, linesize 16 bytes. PID hash table entries: 32 (order 5: 256 bytes) Using 120.000 MHz high precision timer. Dentry cache hash table entries: 2048 (order: 1, 8192 bytes) Inode-cache hash table entries: 1024 (order: 0, 4096 bytes) Memory: 6096k/7808k available (1226k kernel code, 1692k reserved, 181k data, 68k init, 0k highmem) Calibrating delay loop... 239.20 BogoMIPS Mount-cache hash table entries: 512 (order: 0, 4096 bytes) Checking for 'wait' instruction... unavailable. NET: Registered protocol family 16 Can't analyse prologue code at 801411fc 1.parse options inodes 764 block 764 PPP generic driver version 2.4.2 NET: Registered protocol family 24 Using noop io scheduler bcm963xx_mtd driver v1.0 rootfs_addr 0bfc10100 brcmboard: brcm_board_init entry bcm963xx_serial driver v2.0 NET: Registered protocol family 2 IP: routing cache hash table of 512 buckets, 4Kbytes TCP: Hash tables configured (established 512 bind 1024) NET: Registered protocol family 1 NET: Registered protocol family 17 NET: Registered protocol family 8 NET: Registered protocol family 20 VFS: Mounted root (squashfs filesystem) readonly. Freeing unused kernel memory: 68k freed init started: BusyBox v1.00 (2005) multi-call binary Algorithmics/MIPS FPU Emulator v1.5 tmpfs size 262144 1.parse options inodes 773 block 64 BusyBox v1.00 (2005) Built-in shell (msh) Enter 'help' for a list of built-in commands. Loading drivers and kernel modules... atmapi: module license 'Proprietary' taints kernel. blaadd: blaa_detect entry adsl: adsl_init entry Broadcom BCMPROCFS v1.0 initialised Internal PHY Broadcom BCM6338A2 Ethernet Network Device v0.3 Jun 1 2007 12:19:47 Config Internal PHY Through MDIO BCM63xx_ENET: Auto-negotiation timed-out BCM63xx_ENET: 10 MB Half-Duplex (assumed) eth0: MAC Address: 00:14:6C:2B:94:31 pSdramPHY=0xA07FFFF8, 0x5344 0x14882240 AdslCoreHwReset: AdslOemDataAddr = 0xA07FB404 ==> Netgear DM111 ADSL2+ Modem Software Version: 3.30j_A2pB021g.d19b <== pvc2684d: Interface "nas_8_35" created sucessfully pvc2684d: Communicating over ATM 0.8.35, encapsulation: LLC ip_tables: (C) 2000-2002 Netfilter core team ip_conntrack version 2.1 (61 buckets, 0 max) - 368 bytes per conntrack ip_conntrack_pptp version 2.1 loaded ip_nat_pptp version 2.0 loaded ip_conntrack_h323: init ip_nat_h323: initialize the module! ip_conntrack_rtsp v0.01 loading ip_nat_rtsp v0.01 loading ip_nat_pptp version 2.0 unloaded ip_conntrack_pptp version 2.1 unloaded unregistering port 554 board_ioctl: boot complete!
As indicated, you can force CFE to give you a prompt by hitting 'Space', after which you should be able to do some useful things, such as booting an alternate kernel via TFTP or re-flashing a bricked modem.
Firmware Layout
From time to time, Netgear provides firmware images to update the modem (see there). Careful examination of those files leads to a quite common firmware layout, which is used in many SOHO modem/routers. The schema below summarises the global layout of a firmware image :
+---------------------------------+ 0x00000000 | | | Broadcom Header | Fixed size = 256 bytes | | +---------------------------------+ 0x00000100 | | | CFE boot loader | Size in header (cfeLen) | | +---------------------------------+ 0x00000100 + cfeLen | | | SquashFS Root | Size in header (rootfsLen) | | +---------------------------------+ 0x00000100 +cfeLen + rootfsLen | | | Compressed kernel | Size in header (kernelLen) | | +---------------------------------+
The Broadcom header
The Broadcom header is well-defined, and extensively described in the OpenWrt bcmTag.h header file. The following table explains the different fields and reports the respective values in images provided by Netgear :
| Name | Length | Description | FW version 3.28R | FW version 3.29U | FW version 3.30J |
|---|---|---|---|---|---|
| tagVersion | 4 | Header version | 6 | 6 | 6 |
| signature_1 | 20 | Company info | N/A | ? | ? |
| signature_2 | 14 | Additional info | N/A | ? | ? |
| chipId | 6 | Chip Id. | N/A | ? | ? |
| boardId | 16 | Board Id. | RTA 1320 | RTA 1320 | RTA 1320 |
| bigEndian | 2 | Big endian flag | 1 | 1 | 1 |
| totalImageLen | 10 | Size of the image (minus header) | 1613850 | 1617509 | 1630076 |
| cfeAddress | 12 | CFE boot loader address (if not null) | 0xbfc00000 | 0xbfc00000 | 0xbfc00000 |
| cfeLen | 10 | CFE boot loader size | 61992 | 62200 | 62264 |
| rootfsAddress | 12 | Root filesystem address (if not null) | 0xbfc10100 | 0xbfc10100 | 0xbfc10100 |
| rootfsLen | 10 | Root filesystem size | 1085440 | 1085440 | 1097728 |
| kernelAddress | 12 | Kernel address (if not null) | 0xbfd19100 | 0xbfd19100 | 0xbfd1c100 |
| kernelLen | 10 | Kernel size | 466418 | 469869 | 470084 |
| dualImage | 2 | Dual image flag | 0 | 0 | 0 |
| inactive | 2 | Inactive image flag | 0 | 0 | 0 |
| reserved | ? | Reserved | ? | ? | ? |
| imageValidationToken | 20 | Image checksum | 0x0fcde49b | 0xf91c2b95 | 0xa5d3bca7 |
| tagValidationToken | 20 | Header checksum | 0x783295dc | 0xc4fe2f99 | 0x6b937861 |
The contents of the signature_1, signature_2, chipId, and reserved fields are not yet totally clear : the first three fields seem to be unused in firmware versions anterior to 3.30j, but used afterwards, as is the case for the first 48 bytes of the field. Following those bytes, however, it seems we have a significant 4-byte field at offset 50 (whose meaning is unknown to me), and a very important field at offset 54, which is the checksum of the root filesystem and the kernel sections combined -- this checksum will be checked at boot-time, so if you don't include it, the modem will simply refuse to boot.
I will probably have to examine Netgear's host utilities for more details; if you're curious, the full fields contained in version 3.30J are listed below :
$ ./bcm_fw_extract -v cfe-dm111-v330j_a2pb021g
[0x00] Header version: 6
[0x04] Vendor signature (1):
69 80 8e ea 10 c5 ca c8 45 31 9a dc 0f 8b a9 f5
00 e1 6d cb
[0x18] Vendor signature (2):
b4 ae 26 aa a9 bc c8 e5 df 12 d7 b8 5b 2e
[0x26] Chip Id.:
bf 9d f6 c6 ec 5a
[0x2c] Board Id.: RTA1320
[0x3c] Big endian ?: 1
[0x3e] Total img size: 0x0018df7c
[0x48] CFE address: 0xbfc00000
[0x54] CFE size: 0x0000f338
[0x5e] Root FS address: 0xbfc10100
[0x6a] Root FS size: 0x0010c000
[0x74] Kernel address: 0xbfd1c100
[0x80] Kernel size: 0x00072c44
[0x8a] Dual img ?: 0
[0x8c] Inactive img ?: 0
[0x8e] Reserved:
e7 cf c5 35 8d 5e c1 2e 63 34 bf 70 6c 57 19 c1
b2 31 5f f0 7b b9 f4 1c 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 35 00 00 00 7f 59 32 a6 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00
- RK checksum (0x36): 0x7f5932a6
[0xd8] Header checksum: 0x6b937861
[0xec] Image checksum: 0xa5d3bca7
The image checksum is a standard 1-complemented CRC32 computed over the whole image, header excluded. As for the header checksum, it is computed over the header as whole, itself excluded. Those checksums are verified by CFE and/or the HTTP flash utilities, so don't forget to update them should you want to modify some parts of the firmware image.
Using the information contained in the header, we can easily extract the different sections of the image. I wrote a little tool that does just that -- it is available in the tools directory. Example usage :
$ ./bcm_fw_extract -a -d 3.30J cfe-dm111-v330j_a2pb021g [0x00] Header version: 6 [0x2c] Board Id.: RTA1320 [0x3c] Big endian ?: 1 [0x3e] Total img size: 0x0018df7c [0x48] CFE address: 0xbfc00000 [0x54] CFE size: 0x0000f338 [0x5e] Root FS address: 0xbfc10100 [0x6a] Root FS size: 0x0010c000 [0x74] Kernel address: 0xbfd1c100 [0x80] Kernel size: 0x00072c44 [0x8a] Dual img ?: 0 [0x8c] Inactive img ?: 0 Dumping Broadcom header... (offset 0x00000000, sz 0x00000100, crc 0x8a24fb98) done. Dumping CFE boot loader... (offset 0x00000100, sz 0x0000f338, crc 0x4991f98f) done. Dumping root filesystem... (offset 0x0000f438, sz 0x0010c000, crc 0xdf4aeffa) done. Dumping kernel... (offset 0x0011b438, sz 0x00072c44, crc 0x0568def0) done.
The CFE bootloader
CFE stands for Common Firmware Environment and is generally the bootloader used on Broadcom boards. The sources for recent versions are provided on Broadcom's website, but the CFE embedded in Netgear routers (at least in the DM111P) is a special version customised to be able to boot an LZMA-compressed kernel (see below). The source for the modified version does not seem to be available, as Netgear's sources only contain binary versions of the CFE.
We can however suppose that it contains already published code -- running strings on the extracted bootloader reveals interesting messages, which can be found in a published 2.6.8.1 kernel patch :
$ strings -12 3.30J/bootldr.bin
PANIC: out of memory!
!Error allocating memory for LitDecoder m_Coders!
!Error in allocating memory for reverseBitTreeDecoder2!
xError in allocating memory for bitTreeDecoder!
Error in allocating memory for reverseBitTreeDecoder!
0CodeReal: invalid data
Since the above-mentioned patch is GPL code, it seems likely that Netgear infringes the GPL by not publishing source code for the modified bootloader they are using.
The root filesystem
The filesystem image is a modified SquashFS image, which is LZMA-compressed (cf. the SquashFS LZMA homepage for more information.). However it seems the SquashFS version used by Netgear is not compatible with the standard tools from the SquashFS LZMA project :
$ unsquashfs -v | head -n 1 unsquashfs version 1.5 (2007/10/31) $ unsquashfs -s 3.30J/rootfs.bin Reading a different endian SQUASHFS filesystem on 3.30J/rootfs.bin Can't find a SQUASHFS superblock on 3.30J/rootfs.bin
The message regarding endianness being purely informative, we have to conclude that Netgear is using a different SquashFS and/or LZMA implementation. Fortunately, the kernel patch mentioned above gives us a hint at the LZMA implementation that is being used. Quickly patching unsquashfs to use this implementation yields something functional (thanks to the fine OpenBox4 folks for what appears to be the original implementation) :
$ ./lzma_unsquash -s 3.30J/rootfs.bin Reading a different endian SQUASHFS filesystem on 3.30J/rootfs.bin Found a valid big endian SQUASHFS 2:0 superblock on 3.30J/rootfs.bin. Creation or last append time Fri Jun 1 06:23:29 2007 Filesystem is not exportable via NFS Inodes are decompressed Data is compressed Check data is not present in the filesystem Fragments are not present in the filesystem Always_use_fragments option is not specified Duplicates are removed Filesystem size 1071.33 Kbytes (1.05 Mbytes) Block size 65536 Number of fragments 0 Number of inodes 351 Number of uids 2 Number of gids 1
Combining this with the image extraction code above, we are now able to extract the entire filesystem :-) :
$ ./bcm_fw_extract -f -x -d 3.30J cfe-dm111-v330j_a2pb021g [0x00] Header version: 6 [...] Dumping root filesystem... (offset 0x0000f438, sz 0x0010c000, crc 0xdf4aeffa) done. Extracting root filesystem... done. $ ls -l 3.30J/squashfs-root total 44 drwxr-xr-x 2 sjv sjv 4096 2007-06-01 06:23 bin drwxr-xr-x 2 sjv sjv 4096 2007-06-01 06:14 CVS drwxr-xr-x 2 sjv sjv 4096 2007-06-01 06:23 dev drwxr-xr-x 10 sjv sjv 4096 2007-06-01 06:23 etc drwxr-xr-x 3 sjv sjv 4096 2007-06-01 06:21 lib lrwxrwxrwx 1 sjv sjv 11 2008-03-23 20:07 linuxrc -> bin/busybox drwxr-xr-x 2 sjv sjv 4096 2007-06-01 06:23 mnt drwxr-xr-x 2 sjv sjv 4096 2007-06-01 06:23 proc drwxr-xr-x 2 sjv sjv 4096 2007-06-01 06:23 sbin drwxr-xr-x 3 sjv sjv 4096 2007-06-01 06:23 usr drwxr-xr-x 2 sjv sjv 4096 2007-06-01 06:23 var drwxr-xr-x 2 sjv sjv 4096 2007-06-01 06:21 webs
The kernel
Examining the kernel image reveals that it is LZMA-compressed, and that there is a Broadcom-specific header prepended to it. The format of this header is quite simple (and originally described at SkayaWiki) : a 4-byte value representing the address at which the kernel will be loaded, a second 4-byte value holding the kernel entry point, i.e. the code that gets executed after decompression, and a third value being the size of the compressed kernel (header stripped). In our case, the header yields the following :
$ hexdump -C 3.30J/kernel.bin | head -n 5 00000000 80 01 00 00 80 17 10 18 00 07 2c 38 5d 00 00 40 |..........,8]..@| 00000010 00 00 00 6f fd ff ff a3 b7 7f 73 46 51 6a 69 37 |...o......sFQji7| 00000020 2f 2a 4c aa 3e ad ff b2 20 92 3f 7a 37 87 f9 f1 |/*L.>... .?z7...| 00000030 b9 c0 6b cd 4e 2c 90 0e e0 d2 49 1b da a9 d7 5d |..k.N,....I....]| 00000040 31 8c a1 b8 59 4b d2 c4 1e 36 29 0e 81 8d 81 5f |1...YK...6)...._|
This means that the bootloader will load the kernel at 0x80010000, and then jump to 0x80171018, which is fully confirmed by the console output above :
[...] Auto run second count down(before hit space key): Code Address: 0x80010000, Entry Address: 0x80171018 Decompression OK! Entry at 0x80171018 Closing network. Starting program at 0x80171018 Linux version 2.6.8.1 (compiled by michaelc) (gcc version 3.4.2) #1 Fri Jun 1 12:20:59 CST 2007 [...]
Since we now have a functioning LZMA decompressor (note that current LZMA utils do *not* seem to be able to decompress this image), we can extract the kernel image :
$ ./bcm_fw_extract -k -x -d 3.30J cfe-dm111-v330j_a2pb021g [0x00] Header version: 6 [...] Dumping kernel... (offset 0x0011b438, sz 0x00072c44, crc 0x0568def0) done. Extracting kernel... (sz 0x00180015) done. $ ls -l 3.30J/vmlinux -rw-r----- 1 sjv sjv 1572885 2008-03-23 20:07 3.30J/vmlinux
This decompressed kernel is a stripped version of a standard MIPS32 ELF vmlinux image. As for booting a kernel directly via TFTP, I'm still investigating : it seems CFE does recognise the header, but somehow fails at decompression -- this needs further investigation.