COMPARING THE MULTITHREADED FLOATING POINT PERFORMANCE OF THE RASPBERRY PI 400 WITH LENOVO THINKPAD ULV NOTEBOOKS USING POVRAY BENCHMARK
Christian Koot, 12/27/2020
Right from the start, the stated goal of the Raspberry Pi Foundation was to promote the teaching of basic computer science in schools and developing countries [1][2]. However, earlier editions of the Raspberry Pi did not only reach the primarily intended audience but became increasingly popular among hobbyists in fields like robotics and home automation and even saw some commercial applications [3]. This was largely due to the easily programmable GPIO interface, which makes it easy to operate a plethora of actuators and sensors [4].
Over the years, the hardware has become much more powerful, including features such as RAM sizes up to 4 (Pi 400) or 8 GB (Pi 4), the support of the ARMv8 64 bit instruction set (ARM64) as well as the successive introduction (and acceleration) of more powerful means to communicate such as Ethernet, Wifi, and Bluetooth [5][6].
In the same way, the available software, which has always been a stronghold of the Raspberry Pi (compared with other single-board computers such as Banana Pi and Odroid) for the completeness of hardware support, has ripened impressively. Among others, Raspberry Pi OS (armhf) and Ubuntu (ARM64) stick out with offering out-of-the-box hardware support on fully-featured Linux desktop systems with vast software repositories providing easy access to a wide range of application software [7][8].
Both the evolution of hardware and software make the Pi 4 and Pi 400 (more than ever before) ready to serve as a desktop replacement. This is especially true for the Pi 400 which brings an integrated keyboard and comes at a remarkably low price tag of $ 70 (or $ 100 for the kit version which includes a power supply, a 16 GB micro SD card, a computer mouse, an HDMI cable and a printed manual) [9]. When using the kit version, all it needs to create a fully productive system is a TV with an HDMI connector and a Wifi network providing internet access.
Just recently, the ARM architecture in general has seen a trend to extending its scope of application. Being traditionally confined to mobile devices for its relatively low power consumption, nowadays, an increasing number of ARM-based designs is tailored to more demanding workloads in desktop, server, and cloud computing. Most notably are the Apple M1 and Amazon Graviton2 - both of which demonstrate perfomance and efficiency levels that seriously match (if not outmatch) their x86-64 rivals [10][11].
The Pi 400 certainly is less powerful by far (and much cheaper, too). But if you aim at using it as a desktop replacement, the comparison to the popular class of notebooks with ULV (ultra-low voltage) CPUs regarding performance might by quite interesting.
Generally, benchmarking is a complex task in which multiple aspects of performance must be carefully measured and weighted. By its very nature (a hobbyist's project on a boring boxing day), this paper cannot do that. Instead, the multithreaded floating point performance will be assessed isolatedly using one single benchmark. The POVray benchmark offers a simple approach here, since POVray 3.7 automatically uses as many threads as the CPU provides in virtual cores for the trace calculations creating severe CPU workload [12].
METHODS
Data was collected and written to a plain CSV file by a simple Python (i.e. CPython 3.8.6) script (fig. 1) based on the standard library's subprocess module running n = 10 consecutive passes of POVray benchmark using POVray 3.7.0.8 on Ubuntu 20.10 (AMD64/ARM64, patch level 12/25/2020) with the packaged benchmark file on this eclectic selection of machines [13][14]:
- Raspberry Pi 400 (2020, Broadcom BCM2711C0)
- Lenovo ThinkPad E595 (2020, 20NF-S0NG00, AMD Ryzen 7 3700U)
- Lenovo ThinkPad L390 (2018, 20NR-001LGE, Intel Core i5-8265U)
- Lenovo ThinkPad 13 (2017, 20J2-S00G00, Intel Core i7-7500U)
- Lenovo ThinkPad E320 (2011, NWY5LGE, Intel Core i3-2330U)
Fig. 1: Python script for collecting benchmark data
The notebooks were operated with a live system of Ubuntu 20.10 AMD64 created by the dus version of mkusb 12.6.6 on some random 16 GB Toshiba USB memory stick I happened to find on my desk [15][16]. The Pi 400 operating system resided on a 32 GB Samsung EVO+ micro SD card. However, the obvious difference in I/O performance is not expected to affect POVray benchmark results, since it does not make notable use of I/O during progress.
Data collection was conducted at an ambient temperature of 22 °C (72 °F). The machines were cold and freshly booted directly before the benchmarking started.
POVray benchmark yields several metrics (photon time, radiosity time, trace time). Since trace time accounts for the largest share (by far) of the computation time, it is used as the relevant variable for this paper.
Trace time (in seconds) is reported for every individual benchmark pass. Minimum, average, and maximum trace times are reported per benchmarked machine.
Raw data from the CSV file was imported into LibreOffice 6.0.7.3. From the imported data, the machines' respective minimum, average and maximum pass times were calculated. Moreover, the relative performance (in %) of the Pi 400 versus the respective benchmarked machine was calculated using the formula in fig. 2.
Fig. 2: Formula for relative performance of Pi 400
RESULTS
Fig. 3 depicts the average trace times per machine. Tab. 1 reports minimum, average, and maximum trace times per machine. The Excel spreadsheet contains the raw data collected by the Python script and the calculated metrics.
Fig. 3: Bar graph of average trace times
Tab. 1: Min. / avg. / max. trace times and relative performance of Pi 400
Link: Raw data and calculated metrics (XLSX)
DISCUSSION
In contrast to the reputation of earlier Raspberry Pi generations for offering limited performance ("good enough") at very low price tags, the Pi 400 does surprisingly well in the POVray benchmark versus the benchmarked ULV notebooks:
Reaching almost 30 % of the multithreaded floating point performance of a modern ULV notebook powered by an AMD Ryzen 7 3700U (as measured by POVray benchmark) might seem sluggish at first glance, but is not. In comparison with older ULV notebooks, which are still perfectly suited for everyday office workloads, the Pi 400 looks far better: It delivers about two thirds of the performance of a 2017 ThinkPad 13 with an Intel Core i7-7500U and drastically outperforms a 2011 ThinkPad Edge E320 (Intel Core i3-2330U). Furthermore, the performance drop over the benchmark runs indicates a much lower extent of thermal throttling for the Pi 400 under constant high CPU load than users experienced with the predecessors. The aluminum plate, the Pi 400 uses as a passive heat sink proves to be a simple yet capable cooling solution. Overall, all benchmarked machines show signs of progressive thermal throttling indicated by increasing run times of the benchmark passes. The only machine showing pronounced cycles of higher and lower performance was the ThinkPad Edge E320.
Of course, the methods of this benchmark comparison are massively limited by only employing one isolated benchmark measure which merely provides a picture of multithreaded floating point performance. Other important aspects, such as single-thread, integer, I/O, and video decoding performance are left aside by the chosen approach.
Nevertheless, the results support the Raspberry Pi 400's ability to serve as a suitable desktop replacement for office, internet, and educational workloads in settings where the price tag matters: These include computer labs at schools (where funding is a typical challenge) or the use as a personal computer for socially disadvantaged groups, providing them with a multitude of opportunities for digital participation, which otherwise simply would not exist.
REFERENCES
[1] https://bbc.co.uk/[...]
[2] https://www.electronicsweekly.com/[...]
[3] https://magpi.raspberrypi.org/[...]
[4] https://www.raspberrypi.org/[...]
[5] https://www.raspberrypi.org/[...]
[6] https://www.raspberrypi.org/[...]
[7] https://www.raspberrypi.org/[...]
[8] https://ubuntu.com/[...]
[9] https://www.raspberrypi.org/[...]
[10] https://www.theguardian.com/[...]
[11] https://www.zdnet.com/[...]
[12] https://www.povray.org/
[13] https://www.python.org/
[14] https://docs.python.org/[...]
[15] https://www.howtogeek.com/[...]
[16] https://help.ubuntu.com/[...]