AppEngine on MacOS is a CPU Hog: Solve This Problem with Another Python Native Extension Module

(this is a repost of a medium article of which I am the author)

Hey, have you read this gorgeous article from Filippo Valsorda about how to write a Python extension in Go? You definitely should.
When I first read it, I was searching for a solution to a problem affecting all the MacBooks in our open space. You have this very same problem if you develop AppEngine applications on MacOS X. You should notice it when you launch dev_appserver.py and your CPU cooler starts going crazy. At this point, you can feel a warm breeze flowing out of your laptop and your SSD starts to age faster than it should.

Analysis of the problem

But what is the problem, exactly? Well, it resides in the mtime_file_watcher.py part of the Google Cloud SDK. This is the default file watcher for MacOS X. The file watcher is the component that watches your project source code and restarts a module to reload the modified code. On Linux, the default file watcher is the inotify_file_watcher.py so you won’t experience that warm breeze unless you don’t specify–use_mtime_file_watcher=yes as a dev_appserver.py option. If you are desperate, you can also disable the watcher entirely –automatic_restart=no but soon you will be even more desperate. Another way to mitigate this phenomenon is to fine tune the watched paths with –watcher_ignore_re=”.*/app/client/.*” but honestly, when you have more than one module to run this will not produce the effects you would expect.

When I first started wandering the net searching for a solution, I found some alternatives. The most impressive article was from zeekay. In its implementation, he was using another fsevents module in Python. I have tried to use it, but it segfaulted on my MacOS. It’s C code, and I didn’t feel like rebuilding and and debugging it. I have to admit that I am more interested in Go than ol’ school C (best TIOBE language of 2017 and my favorite in 2002–2004, though).

Enter Filippo. His article sufficiently sparked my curiosity and was complete enough to get my extension running. He covers the structure of a Python extension written in Go, with the minimal C infrastructure to expose it to the Python interpreter. He shows how to parse two integers and how to return them to the caller.

My aim was to parse a list of paths and call a Python callback when an event was raised on one of the given paths. Returning the modified file and the action flags (Created, Modified, Deleted and so on). I wanted to do that in a parallel way (ie. call the callback immediately, once the event is raised. goroutines are great for that.)

What you can learn from my experience

• Go has a great development ecosystem. The most significative part of the go tooling comes from the compiler itself, but you will certainly find an extension for your favorite editor too.
• Go has pointers. After 7 years without them, it’s a bit weird to get to know them again. The go compiler (and its syntax checker) helped me a lot through this step.
  Parsing a list of strings is a bit different from parsing two integers. This was my first segfault, and what a pleasant surprise to see that a go binary gives you the full annotated stack when this happens!
• (Filippo already pointed that out, thank you) C variadic functions are not accessible from cgo. Variadic means “with a variable number of parameters at its end” — like *args in Python)
• Useful C macros are not visible from Go. For example, I’m thinking of Py_?DECREF orPy_?INCREF. So, you have to write a wrapper function in your C portion of code. C purists will probably cry over this, but hey, it works 🙂
• If you use goroutines in Go, you will have called the Py_InitThreads during the module initialization, in the C portion of the extension. This was my second segfault. The stack pointed out my C function, so at least I have had to debug the C code and learn a little bit of LLDB. Once I discovered which Python function was responsible for the segfault (i.e. PyGILState_Release) a little googling gave me the solution.
• In the case of an eventual porting of the extension code to Python 3, some clever developers have put in place a practical macro. (I have borrowed mine from Malthe Borch, and I found similar ones elsewhere.)

Once the POC was complete, there was one more obstacle to avoid: packaging. That means getting rid of “it works on my machine!” and making it a *product*. Enter the Python scripts to backup, replace (and eventually restore) the mtime file watcher that comes with AppEngine.

Result

On my machine, the main Python process started by the dev_appserver.py was consuming 156% CPU, constantly. In early January, I was about to ask my manager for a new MacBook.
Today, the code in the https://github.com/nilleb/fsevents-watcher repository allows Python to consume only 0.3% CPU. And, of course, the AppEngine application modules are being reloaded when needed.
What’s next
I would really like to write some unit tests, and understand a little more about what makes the Python interpreter delivered with the OS binary incompatible with the home brew one (even if they share the same major.minor version numbers). Once this is done, I will then perhaps build a pip package and publish it on pypi.org.

Caveats

If you update the Google Cloud components, you have to re-replace mtime_file_watcher.py with the scripts provided with the module.

Special thanks

This project was a fruit of the first LumApps Polish Week, held from January 22 to January 26. Special thanks to Lou for igniting and supporting this event.

What’s a polish week? That’s a week during which a part of LumApps takes a break from delivery deadlines, in order to improve the collective health. Everyone is free to suggest a problem and volunteer to solve it.