47

I have tried using the line_profiler module for getting a line-by-line profile over a Python file. This is what I've done so far:

1) Installed line_profiler from pypi by using the .exe file (I am on WinXP and Win7). Just clicked through the installation wizard.

2) Written a small piece of code (similar to what has been asked in another answered question here).

from line_profiler import LineProfiler def do_stuff(numbers): print numbers numbers = 2 profile = LineProfiler(do_stuff(numbers)) profile.print_stats()

3) Run the code from IDLE/PyScripter. I got only the time. 

Timer unit: 4.17188e-10 s

How do I get full line-by-line profile over the code I execute? I have never used any advanced Python features like decorators, so it is hard for me to understand how shall I use the guidelines provided by several posts like here and here.

Improve this question

6 Answers 6

Reset to default
91

This answer is a copy of my answer here for how to get line_profiler statistics from within a Python script (without using kernprof from the command line or having to add @profile decorators to functions and class methods). All answers (that I've seen) to similar line_profiler questions only describe using kernprof.

The line_profiler test cases (found on GitHub) have an example of how to generate profile data from within a Python script. You have to wrap the function that you want to profile and then call the wrapper passing any desired function arguments.

from line_profiler import LineProfiler import random def do_stuff(numbers): s = sum(numbers) l = [numbers[i]/43 for i in range(len(numbers))] m = ['hello'+str(numbers[i]) for i in range(len(numbers))] numbers = [random.randint(1,100) for i in range(1000)] lp = LineProfiler() lp_wrapper = lp(do_stuff) lp_wrapper(numbers) lp.print_stats()

Output:

Timer unit: 1e-06 s Total time: 0.000649 s File: <ipython-input-2-2e060b054fea> Function: do_stuff at line 4 Line # Hits Time Per Hit % Time Line Contents ============================================================== 4 def do_stuff(numbers): 5 1 10 10.0 1.5 s = sum(numbers) 6 1 186 186.0 28.7 l = [numbers[i]/43 for i in range(len(numbers))] 7 1 453 453.0 69.8 m = ['hello'+str(numbers[i]) for i in range(len(numbers))]

Adding Additional Functions to Profile

Also, you can add additional functions to be profiled as well. For example, if you had a second called function and you only wrap the calling function, you'll only see the profile results from the calling function.

from line_profiler import LineProfiler import random def do_other_stuff(numbers): s = sum(numbers) def do_stuff(numbers): do_other_stuff(numbers) l = [numbers[i]/43 for i in range(len(numbers))] m = ['hello'+str(numbers[i]) for i in range(len(numbers))] numbers = [random.randint(1,100) for i in range(1000)] lp = LineProfiler() lp_wrapper = lp(do_stuff) lp_wrapper(numbers) lp.print_stats()

The above would only produce the following profile output for the calling function:

Timer unit: 1e-06 s Total time: 0.000773 s File: <ipython-input-3-ec0394d0a501> Function: do_stuff at line 7 Line # Hits Time Per Hit % Time Line Contents ============================================================== 7 def do_stuff(numbers): 8 1 11 11.0 1.4 do_other_stuff(numbers) 9 1 236 236.0 30.5 l = [numbers[i]/43 for i in range(len(numbers))] 10 1 526 526.0 68.0 m = ['hello'+str(numbers[i]) for i in range(len(numbers))]

In this case, you can add the additional called function to profile like this:

from line_profiler import LineProfiler import random def do_other_stuff(numbers): s = sum(numbers) def do_stuff(numbers): do_other_stuff(numbers) l = [numbers[i]/43 for i in range(len(numbers))] m = ['hello'+str(numbers[i]) for i in range(len(numbers))] numbers = [random.randint(1,100) for i in range(1000)] lp = LineProfiler() lp.add_function(do_other_stuff) # add additional function to profile lp_wrapper = lp(do_stuff) lp_wrapper(numbers) lp.print_stats()

Output:

Timer unit: 1e-06 s Total time: 9e-06 s File: <ipython-input-4-dae73707787c> Function: do_other_stuff at line 4 Line # Hits Time Per Hit % Time Line Contents ============================================================== 4 def do_other_stuff(numbers): 5 1 9 9.0 100.0 s = sum(numbers) Total time: 0.000694 s File: <ipython-input-4-dae73707787c> Function: do_stuff at line 7 Line # Hits Time Per Hit % Time Line Contents ============================================================== 7 def do_stuff(numbers): 8 1 12 12.0 1.7 do_other_stuff(numbers) 9 1 208 208.0 30.0 l = [numbers[i]/43 for i in range(len(numbers))] 10 1 474 474.0 68.3 m = ['hello'+str(numbers[i]) for i in range(len(numbers))]

NOTE: Adding functions to profile in this way does not require changes to the profiled code (i.e., no need to add @profile decorators).

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

Good answer! But I actually find the decorators a good way to control profiling. Is there a way to use line_prof with decorators within-script (without kernprof)?
I'm not 100% this works. I haven't tested it yet. Thanks to tdube for the line_profiler code. All I did was convert an existing decorator I had to hopefully work with LineProfiler. EDIT: formatting was messed up because I can't figure out how to format code in comments. Here's a Pastebin.
That's what I'm using currently in my Django code (where I can't use command line kernprof): stackoverflow.com/a/68163807/1937033
Is it possible to wrap the entire script? I mean I could technically make it one big function but that seems pretty overkill
21

Just follow Dan Riti's example from the first link, but use your code. All you have to do after installing the line_profiler module is add a @profile decorator right before each function you wish to profile line-by-line and make sure each one is called at least once somewhere else in the code—so for your trivial example code that would be something like this:

example.py file:

@profile def do_stuff(numbers): print numbers numbers = 2 do_stuff(numbers)

Having done that, run your script via the kernprof.py✶ that was installed in your C:\Python27\Scripts directory. Here's the (not very interesting) actual output from doing this in a Windows 7 command-line session:

> python "C:\Python27\Scripts\kernprof.py" -l -v example.py 2 Wrote profile results to example.py.lprof Timer unit: 3.2079e-07 s File: example.py Function: do_stuff at line 2 Total time: 0.00185256 s Line # Hits Time Per Hit % Time Line Contents ============================================================== 1 @profile 2 def do_stuff(numbers): 3 1 5775 5775.0 100.0 print numbers

You likely need to adapt this last step—the running of your test script with kernprof.py instead of directly by the Python interpreter—in order to do the equivalent from within IDLE or PyScripter.

✶Update

It appears that in line_profiler v1.0, the kernprof utility is distributed as an executable, not a .py script file as it was when I wrote the above. This means the following now needs to used to invoke it from the command-line:

> "C:\Python27\Scripts\kernprof.exe" -l -v example.py
Improve this answer

8 Comments

You're welcome. BTW, if you want to run the script normally (without kernprof.py) you'll need to remove the @profile decorator call(s) or define your own dummy one: e.g. profile = lambda f: f at the beginning of the file.
This isn't working for me because I don't seem to be able to call kernprof.py. I used pip install to get line_profiler and a file called kernprof is there but it doesn't have the .py extension, do you know why this would be? Thank you.
@Alex: No idea...could be a bad install. Try it again and it that doesn't work do it manually without pip by downloading the module's source from pypi.
@Alex kernprof doesn't have to have .py extension to be executable. Mine installs inside a venv and runs fine as simply kernprof.
@martineau Do you know why the original code didn't gather statistics? It is a minimal example and the invocation "looks ok" to me. Seems to run fine when one replaces LineProfiler(..) with cProfile.Profile(...).
|
9

Found a good use to line_profiler using decorator i.e. @profile that worked for me:

def profile(func): from functools import wraps @wraps(func) def wrapper(*args, **kwargs): from line_profiler import LineProfiler prof = LineProfiler() try: return prof(func)(*args, **kwargs) finally: prof.print_stats() return wrapper

Credits to: pavelpatrin

Improve this answer

Comments

6

load the line_profiler and numpy

%load_ext line_profiler import numpy as np

define a function for example:

def take_sqr(array): sqr_ar = [np.sqrt(x) for x in array] return sqr_ar

use line_profiler to count the time as follows:

%lprun -f take_sqr take_sqr([1,2,3])

the output looks like this:

Timer unit: 1e-06 s Total time: 6e-05 s File: <ipython-input-5-e50c1b05a473> Function: take_sqr at line 1 Line # Hits Time Per Hit % Time Line Contents ============================================================== 1 def take_sqr(array): 2 4 59.0 14.8 98.3 sqr_ar = [np.sqrt(x) for x in array] 3 1 1.0 1.0 1.7 return sqr_ar
Improve this answer

3 Comments

Welcome to Stack Overflow! Code-only answers are not particularly helpful. Please include a brief description of how this code solves the problem.
Thank you! I just edited the response. I hope you find it clearer now.
I used just this way but I would like to suppress that output because I don't need that by now (only need output during debug)
4

If you're using PyCharm, you can also take a look at https://plugins.jetbrains.com/plugin/16536-line-profiler

It's a plugin I created that allows you to load and visualize line profiler results into the PyCharm editor.

Improve this answer

1 Comment

Do you have any manual on how to remove the values once you want to retake coding?, congrats by the way, this is pretty cool.
2

Just an addition to @Lhenkel answer. This is a decorator for async functions

def async_profile(func): """line profiler for an async funciton""" from functools import wraps @wraps(func) async def wrapper(*args, **kwargs): from line_profiler import LineProfiler prof = LineProfiler() try: return await prof(func)(*args, **kwargs) finally: prof.print_stats() return wrapper

To use these decorators with methods read this answer

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.