Profiling

GeoUtils has a built-in profiling tool, that can be used to provide more insight on the memory and time use of a function if needed.

Warning

The profiling functionalities rely on psutil and plotly, which can be installed manually or using Geoutils development dependencies.

With the profiling activated with the graphs output, two kinds of .HTML graphs will be created by default :

• an icicle graph time_graph.html, showing the time spent in each step of the entire process

• a graph memory_[function].html for each decorated functions used, showing the memory consumption of GeoUtils at regular intervals during the execution

Configuration and parameters

GeoUtils’s profiling configuration works just like a pipeline step. It is executed only if at least one of this two parameters is set to True :

Name	Description	Type	Default value	Required
save_graphs	Save the default graphs generated	bool	False	No
save_raw_data	Save the raw data on calls as a .pickle file	bool	False	No

Example of initialization:

After this, if save_graphs or save_raw_data are True, every profiled function will be studied by the profiler.

Saved profiling data

The function generate_summary(output) saves the profiling information in the output path directory given in input (output_profiling if not specified).

• When save_graphs is enabled, the graphs as listed as before are exported.

• when save_raw_data is enabled, GeoUtils saves all stored information as a .pickle file containing a DataFrame with the following structure:

Name	Description
level	Depth of the function call in the profiling stack
uuid_function	Unique universal identifier (UUID) of the function call
name	Understandable name given to the function call
uuid_parent	UUID of the “parent” call (call that was running when this call was made)
time	Time (in seconds) it took to execute the function
call_time	Timestamp (in seconds) at which the call was made
memory	Either None or a list of (timestamp memory) tuples representing memory consumption (in megabytes) at each timestamp during the function execution

If no profiled function has been called, generate_summary(output) generate no output.

The profiled functions

Currently, some processes are already profiled by GeoUtils with a memory consumption report each 0.05 seconds.

• Raster(), get_stats(), crop(), icrop(), reproject(), interp_points() and subsample()

• PointCloud(), get_stats(), subsample() and grid()

• Vector(), crop(), reproject(), rasterize()

• all existing geo-transformations like _reproject(), _crop() and _translate() for Raster, PointCloud and Vector

• subsample_array() and _statistics() in Geoutils.stats

• _interp_points() in Geoutils.interface.interpolate

Modifying the profiled functions

To profile other functions and add them to the summary graphs and data, simply add the @profile_tool decorator before them, providing a descriptive name.

If you also want to track memory usage over time for a specific function call, set memprof=True in the decorator. If the function is too fast (or slow) for the default memory sampling interval, you can modify it with interval (in seconds).

Name	Description	Type	Default value	Required
name	Name of the function in the report	str		Yes
interval	Memory sampling interval (seconds)	float	0.005	No
memprof	Whether to profile the memory consumption or not	bool	False	No

Output graphs example

Here are two examples of graphs with a personal profiled function my_program, containing the computation of a few attributes and a co-registration:

You can experiment with these graphs here : memory_my_program.html and time_graph.html.