24 posts tagged “profiling”
2026
Includes performance optimizations written by Claude Opus 5.5 after I fed it this Python 3.15 profiling report generated by GPT-6 Astra.
Hello Python 3.15, goodbye 3.10, and now I can use 3.11
Python 3.15 is here! Here’s the essential What’s new in Python 3.15 guide, this year curated by 3.14 and 3.15 release manager Hugo van Kemenade. As someone who builds open source software that works in every supported Python version, I get to say goodbye to Python 3.10 and start embracing features from Python 3.11.
[... 1,198 words]I put the new Python 3.15 profiler through its paces by having GPT-6 Astra Extra High in Codex run it against my new web-edit app:
I need a Python 3.15 flame graph and heatmap demo of the "web-edit" app from PyPI - research first then make me those demos
Here's the resulting example flame graph and heatmap.
2025
python-importtime-graph.
I was exploring why a Python tool was taking over a second to start running and I learned about the python -X importtime feature, documented here. Adding that option causes Python to spit out a text tree showing the time spent importing every module.
I tried that like this:
python -X importtime -m llm plugins
That's for LLM running 41 different plugins. Here's the full output from that command, which starts like this:
import time: self [us] | cumulative | imported package
import time: 77 | 77 | _io
import time: 19 | 19 | marshal
import time: 131 | 131 | posix
import time: 363 | 590 | _frozen_importlib_external
import time: 450 | 450 | time
import time: 110 | 559 | zipimport
import time: 64 | 64 | _codecs
import time: 252 | 315 | codecs
import time: 277 | 277 | encodings.aliases
Kevin Michel built this excellent tool for visualizing these traces as a treemap. It runs in a browser - visit kmichel.github.io/python-importtime-graph/ and paste in the trace to get the visualization.
Here's what I got for that LLM example trace:
As you can see, it's pretty dense! Here's the SVG version which is a lot more readable, since you can zoom in to individual sections.
Zooming in it looks like this:
2024
Did you know about Instruments? (via) Thorsten Ball shows how the macOS Instruments app (installed as part of Xcode) can be used to run a CPU profiler against any application - not just code written in Swift/Objective C.
I tried this against a Python process running LLM executing a Llama 3.1 prompt with my new llm-gguf plugin and captured this:
2022
Making Code Faster. Tim Bray’s detailed guide to using the Go profiler.
Compiling Black with mypyc (via) Richard Si is a Black contributor who recently obtained a 2x performance boost by compiling Black using the mypyc tool from the mypy project, which uses Python type annotations to generate a compiled C version of the Python logic. He wrote up this fantastic three-part series describing in detail how he achieved this, including plenty of tips on Python profiling and clever optimization tricks.
2021
How I cut GTA Online loading times by 70% (via) Incredible debugging war story: t0st was fed up of waiting six minutes (!) for GTA Online to load on their PC, so they used a host of devious debugging tricks to try and figure out what was going on. It turned out the game was loading a 10MB JSON file detailing all of the available in-game purchases, but inefficient JSON parsing meant it was pegging an entire CPU for 4 minutes mainly running the strlen() C function. Despite not having access to source code or debugging symbols t0st figured out the problem and managed to inject a custom DLL that hooked some internal functions and dropped load times down from 6m down to to 1m50s!
2019
Using memory-profiler to debug excessive memory usage in healthkit-to-sqlite. This morning I figured out how to use the memory-profiler module (and mprof command line tool) to debug memory usage of Python processes. I added the details, including screenshots, to this GitHub issue. It helped me knock down RAM usage for my healthkit-to-sqlite from 2.5GB to just 80MB by making smarter usage of the ElementTree pull parser.
2018
benfred/py-spy (via) A Python port of Julia Evans’ rbspy profiler, which she describes as “probably better” than the original. I just tried it out and it’s really impressive: it’s written in Rust but has precompiled binaries so you can just run “pip install py-spy” to install it. Shows live output in the terminal while your program is running and also includes the option to generate neat SVG flame graphs.
Build impossible programs. Delightful talk by Julia Evans describing how she went about building a Ruby profiler in Rust despite having no knowledge of Ruby internals and only beginner’s knowledge of Rust.
Using flamegraphs. I really like flamegraphs as a profiling tool—we have support for them baked into our Tikibar debugging toolbar at Eventbrite—but interpreting them isn’t particularly intuitive on first glance. Julia Evans has put together a great explanation of how to read them as part of the documentation for her rbspy Ruby profiler.
2017
How do Ruby & Python profilers work? Julia Evans: “As a precursor to writing a Ruby profiler I wanted to do a survey of how existing Ruby & Python profilers work.”
profiling. “An interactive continuous Python profiler”. This is really neat—simply run “profiling myscript.py” to get an interactive, navigable console-based profile inspector at the end of your script... or run “profiling live-profile mywebverver.py” to see a live, updating profile of a long-running process. Has options for statistical profiling as well, which has a much lower overhead in exchange for a less accurate view of what is going on.
Building a statistical profiler in python. Generating flame graphs of production Python code is surprisingly straight-forward. Brian Pitts built a statistical flame graph profiler into our tikibar debugging tool at Eventbrite and it’s proved extremely useful.
2009
How to find un-indexed queries in MySQL, without using the log (via) Use tcpdump(!) to sniff the MySQL protocol and dump out queries that had the “no index used” bit set.
Building a Better JavaScript Profiler with WebKit. Clever hack from Francisco Tolmasky which solves the problem of JavaScript profilers showing ? as the name of any anonymous functions. He patched the WebKit profiler to look for a displayName attribute on a function and show that as the function name instead.
Heap Dump Analysis. Using jmap to dump the JVM’s memory to disk, then analysing it using the visualvm GUI tool.
Announcing django-viewtools. A really excellent idea—run ./manage.py viewtools --pdb /path/on/site/ to debug a view in your Django project that is raising an error using the Python debugger, or use --profile to run the full request cycle for that URL through the profiler.
2008
Using the New MySQL Query Profiler. Extremely powerful new feature in MySQL 5.0.37. Definitely something for the Django debug toolbar.
Deep Profiling jQuery Apps. Neat plugin from John Resig that monkey-patches most (all?) of the jQuery methods to build up a detailed profile of which methods are being used by a given page.
Debugging Django
I gave a talk on Debugging Django applications at Monday’s inaugural meeting of DJUGL, the London Django Users Group. I wanted to talk about something that wasn’t particularly well documented elsewhere, so I pitched the talk as “Bug Driven Development”—what happens when Test Driven Development goes the way of this unfortunate pony.
The slides [... 1,759 words]2007
YSlow: Bug (fix) in Firebug’s Net Panel. The latest release of the YSlow page analysis plugin (announced at FOWA) also fixes a misleading bug in Firebug’s Net panel.
AJAX Debugging with Firebug. Great Firebug tutorial from creator Joe Hewitt himself. I didn’t know you could trigger profiling from your own code using console.profile() / console.profileEnd().