![[Home page]](PythonPowered.gif "Home page"){kind=small}
| Books | Code | Blog | Python | Author | Train | Find | ©M.Lutz |
For related results, see the following pages:
[February 2009] Python 3.0.1, released in February 2009, speeds up one mode of file input, but does not generally address the other I/O speed concerns described in the preceding section. Specifically, reading files in all-at-once mode, which was the most dramatic slowdown in 3.0.0, has been improved much in 3.0.1, thanks to a buffer allocation scheme rewrite. Essentially, the buffer scheme has been reverted to mimic that in 2.X Python.
However, 3.0.1 does not address any of the other I/O slowdowns mentioned in the previous section, so I am leaving that section's material on this page until a later release. At present, it appears that Python 3.1, due out in June 2009, will attempt to address the remaining I/O speed issues by rewriting the I/O stack in the C language for performance. When available, this may allow 3.0 to approach the I/O speed of 2.X (some reports claim 3.0 I/O runs within 30% of 2.X speed with the change, though this is yet to be verified).
Under the new Python 3.0.1, my benchmark scripts described in the preceding section produce the results in this file. Below is the summary portion of those results for 3.0.1, as well as same results under 3.0.0 (a.k.a 3.0) for comparison. I reran the 3.0.0 test on the same machine used for the 3.0.1 test, so it would correlate more closely: the full comparable 3.0.0 results are available in this file:
# 3.0.0 results [read_byLines_textMode (large.txt=2.80M)] => 3.0 is 44.222 times slower [read_byLines_binaryMode (large.txt=2.80M)] => 3.0 is 83.734 times slower [read_byBlocks_textMode (large.txt=2.80M)] => 3.0 is 55.707 times slower [read_byBlocks_binaryMode (large.txt=2.80M)] => 3.0 is 1.910 times slower [read_allAtOnce_textMode (large.txt=2.80M)] => 3.0 is 55.298 times slower [read_allAtOnce_binaryMode (large.txt=2.80M)] => 3.0 is 131.986 times slower [read_byBlocks_binaryMode (large.bin=12.56M)] => 3.0 is 1.532 times slower [read_allAtOnce_binaryMode (large.bin=12.56M)] => 3.0 is 564.707 times slower [write_byLines_textMode (testIO.out=25.00M)] => 3.0 is 12.212 times slower [write_byLines_binaryMode (testIO.out=25.00M)] => 3.0 is 1.776 times slower [write_byBlocks_textMode (testIO.out=25.00M)] => 3.0 is 3.509 times slower [write_byBlocks_binaryMode (testIO.out=25.00M)] => 3.0 is 2.740 times slower [write_allAtOnce_textMode (testIO.out=25.00M)] => 3.0 is 8.462 times slower [write_allAtOnce_binaryMode (testIO.out=25.00M)] => 3.0 is 0.527 times slower # 3.0.1 results [read_byLines_textMode (large.txt=2.80M)] => 3.0 is 34.401 times slower [read_byLines_binaryMode (large.txt=2.80M)] => 3.0 is 88.652 times slower [read_byBlocks_textMode (large.txt=2.80M)] => 3.0 is 10.254 times slower [read_byBlocks_binaryMode (large.txt=2.80M)] => 3.0 is 1.884 times slower [read_allAtOnce_textMode (large.txt=2.80M)] => 3.0 is 6.638 times slower [read_allAtOnce_binaryMode (large.txt=2.80M)] => 3.0 is 4.283 times slower [read_byBlocks_binaryMode (large.bin=12.56M)] => 3.0 is 1.489 times slower [read_allAtOnce_binaryMode (large.bin=12.56M)] => 3.0 is 10.988 times slower [write_byLines_textMode (testIO.out=25.00M)] => 3.0 is 8.886 times slower [write_byLines_binaryMode (testIO.out=25.00M)] => 3.0 is 2.008 times slower [write_byBlocks_textMode (testIO.out=25.00M)] => 3.0 is 4.095 times slower [write_byBlocks_binaryMode (testIO.out=25.00M)] => 3.0 is 4.189 times slower [write_allAtOnce_textMode (testIO.out=25.00M)] => 3.0 is 5.197 times slower [write_allAtOnce_binaryMode (testIO.out=25.00M)] => 3.0 is 0.820 times slowe
As you can see, the three radically slow read all-at-once tests have improved substantially in 3.0.1 -- from 55, 131, and 564 times slower, to 6, 4, and 10 times slower. This is a major improvement; in fact, the 564 times slower test was so slow in 3.0 that it could not run at all in reasonable time for large files.
On the other hand, these all-at-once input modes are still 4 to 10 times slower in 3.0.1. Further, other input and output modes generally show no significant change. For example, the most common text input mode, line-by-line, is still roughly as slow as it was in 3.0 (34 times slower), and writing output still shows no noteworthy improvement either (4X-8X slower, typically). Reading by-blocks in text mode shows some improvement too (10X slower versus 55X), but this is likely to be an uncommon use case in 3.0 code. Other tests seem too close to differ in 3.0.1 (small variations from run to run are expected and negligable, due to system load, caching, and so on, but these results are typical).
As in the prior section, paring these results down to 3.0 common use cases (reading and writing by-lines for text, and by-blocks for binary) leaves the following results:
# 3.0.0 results [read_byLines_textMode (large.txt=2.80M)] => 3.0 is 44.222 times slower [read_byBlocks_binaryMode (large.txt=2.80M)] => 3.0 is 1.910 times slower [read_allAtOnce_textMode (large.txt=2.80M)] => 3.0 is 55.298 times slower [read_allAtOnce_binaryMode (large.txt=2.80M)] => 3.0 is 131.986 times slower [read_byBlocks_binaryMode (large.bin=12.56M)] => 3.0 is 1.532 times slower [read_allAtOnce_binaryMode (large.bin=12.56M)] => 3.0 is 564.707 times slower [write_byLines_textMode (testIO.out=25.00M)] => 3.0 is 12.212 times slower [write_byBlocks_binaryMode (testIO.out=25.00M)] => 3.0 is 2.740 times slower [write_allAtOnce_textMode (testIO.out=25.00M)] => 3.0 is 8.462 times slower [write_allAtOnce_binaryMode (testIO.out=25.00M)] => 3.0 is 0.527 times slower # 3.0.1 results [read_byLines_textMode (large.txt=2.80M)] => 3.0 is 34.401 times slower [read_byBlocks_binaryMode (large.txt=2.80M)] => 3.0 is 1.884 times slower [read_allAtOnce_textMode (large.txt=2.80M)] => 3.0 is 6.638 times slower [read_allAtOnce_binaryMode (large.txt=2.80M)] => 3.0 is 4.283 times slower [read_byBlocks_binaryMode (large.bin=12.56M)] => 3.0 is 1.489 times slower [read_allAtOnce_binaryMode (large.bin=12.56M)] => 3.0 is 10.988 times slower [write_byLines_textMode (testIO.out=25.00M)] => 3.0 is 8.886 times slower [write_byBlocks_binaryMode (testIO.out=25.00M)] => 3.0 is 4.189 times slower [write_allAtOnce_textMode (testIO.out=25.00M)] => 3.0 is 5.197 times slower [write_allAtOnce_binaryMode (testIO.out=25.00M)] => 3.0 is 0.820 times slowe
And eliminating all-at-once modes to support pathologically large fields leaves these results:
# 3.0.0 results [read_byLines_textMode (large.txt=2.80M)] => 3.0 is 44.222 times slower [read_byBlocks_binaryMode (large.txt=2.80M)] => 3.0 is 1.910 times slower [read_byBlocks_binaryMode (large.bin=12.56M)] => 3.0 is 1.532 times slower [write_byLines_textMode (testIO.out=25.00M)] => 3.0 is 12.212 times slower [write_byBlocks_binaryMode (testIO.out=25.00M)] => 3.0 is 2.740 times slower # 3.0.1 results [read_byLines_textMode (large.txt=2.80M)] => 3.0 is 34.401 times slower [read_byBlocks_binaryMode (large.txt=2.80M)] => 3.0 is 1.884 times slower [read_byBlocks_binaryMode (large.bin=12.56M)] => 3.0 is 1.489 times slower [write_byLines_textMode (testIO.out=25.00M)] => 3.0 is 8.886 times slower [write_byBlocks_binaryMode (testIO.out=25.00M)] => 3.0 is 4.189 times slower
With the omissions, 3.0.1 in common use is still 34X slower for text input, some 50% slower for binary input, and 4X-8X slower for output.
However, I've included these subsets for comparison with the prior section only, as neither may be entirely valid, as discussed earlier. Paring down to expected common 3.0 use cases is somewhat invalid, as some eliminated modes may offer optimization options. Moreover, removing all-at-once cases is even less valid, as all-at-once modes are very commonly used in Python code when file size is predictably reasonable, and sometimes run quicker. As usual, you should consider your own file modes and sizes when analysing performance.
Although the speed-up of the chronically slow all-at-one input cases in 3.0.1 is a much needed improvement, it is a limited optimization. Programmers who must read or write large data sets are still likely better served by Python 2.X, until the wider speedups in the anticipated Python 3.1 release can be benchmarked.
Despite the limited speed-ups, though, the best news here is probably that the issues are being addressed, and will likely become less a factor when 3.1 is released later this year.