Repository navigation
Large regression in memory usage due to the change which allows unicode characters in EmailField (Python 2.7.x) #1832
Description
Version 0.13.0 included a change which allows unicode (non-ascii) characters in the EmailField - 466935e.
This change seemed to have introduced a large regression in the memory usage of the code which uses EmailField and runs under Python 2.7.x.
I imagine that's due to precompiling of the new regular expression used on the EmailField and storing precompiled version in memory and some edge case / bug in CPython 2.7.x (the issue can only be reproduced under 2.7.x, but not 3.x).
I can consistently reproduce this issue inside our project (StackStorm/st2), but I'm also including a very simple script which reproduces it - https://gist.github.com/Kami/1674fcf2e6a2e376c94095307a6b7b65
Script running using mongoengine v0.12.0 (version before the EmailField change was introduced):
(virtualenv) vagrant@ubuntu-xenial:/data/stanley$ python test_mongoengine.py
Python: 2.7.12 (default, Dec 4 2017, 14:50:18)
[GCC 5.4.0 20160609] on Linux-4.4.0-130-generic-x86_64-with-Ubuntu-16.04-xenial
mongoengine: 0.12.0
sleeping...
(virtualenv) vagrant@ubuntu-xenial:/data/stanley$ ps -eo rss,pid,euser,args:100 --sort %mem | grep -i 'test_mongoengine' |grep -v grep | awk '{printf $1/1024 "MB"; $1=""; print }'
18.2969MB 6320 vagrant python test_mongoengine.pyScript running using mongoengine 0.15.3 (latest version, same thing occurs when running it with 0.13.0):
Python: 2.7.12 (default, Dec 4 2017, 14:50:18)
[GCC 5.4.0 20160609] on Linux-4.4.0-130-generic-x86_64-with-Ubuntu-16.04-xenial
mongoengine: 0.15.3
sleeping...
(virtualenv) vagrant@ubuntu-xenial:/data/stanley$ ps -eo rss,pid,euser,args:100 --sort %mem | grep -i 'test_mongoengine' |grep -v grep | awk '{printf $1/1024 "MB"; $1=""; print }'
70.6875MB 6348 vagrant python test_mongoengine.pyAs you can see, the memory usage increases by more than 3 times. Looking at the comment here (466935e#commitcomment-26845075), we are not the one people experiencing this issue.
(This issue was split out from my original comment on the commit here - 466935e#commitcomment-23225459)
Some other related issues / comments:
Activity
Revert a change which causes a big regressions in EmailField memory
bagerard commentedon Jun 16, 2019
Closing this, the lazy evaluation of the regexes addresses the issue for users that aren't using the EmailField. The regex from the EmailField and its lazy-evaluation were inspired by django's EmailField
Happy to reopen it if you see a way to improve this. Note that if you need to use EmailField but don't need the strict validation (which bring slowness and increased memory usage) from the current regex, you can easily subclass EmailField and use a simpler regex