review sandag one zone vs two zone setup performance

[bstabler]

• Goal is to understand performance differences from 1 zone to 2 zone
• Compare model specifications for model implementations
• Tour mode choice slows from 1 zone to 2 zone, why?
• Trip destination submodel slows down as well
• Want to control hardware configuration as well
• @bstabler, @jpn-- compare specs and review logging
• @wusun2 share logfiles, run with same specs across runs, and update example_sandag (by testing on a fork)

[bstabler]

I'll take the lead for this and coordinate with @jpn--. Thanks @esanchez01 for updating the examples. Can you also share the logfiles from your runs?

[esanchez01]

I'll take the lead for this and coordinate with @jpn--. Thanks @esanchez01 for updating the examples. Can you also share the logfiles from your runs?

@bstabler I sent over the location of the 1 & 2 zone logfiles through email.

[bstabler]

@jpn-- and I did some initial review and found the following:

1 - the expression evaluation for the tour_mode_choice is slower for ONE zone versus TWO zone

ONE ZONE

16/06/2021 23:47:30 - INFO - activitysim.core.mem - trace_memory_info tour_mode_choice.work.simple_simulate.chunk_1.eval_nl.eval_utils.add.expression_values rss: 62.41GB used: 122.29 GB percent: 38.2%
16/06/2021 23:47:35 - INFO - activitysim.core.mem - trace_memory_info tour_mode_choice.work.simple_simulate.chunk_1.eval_nl.eval_utils.add.utilities rss: 62.51GB used: 122.28 GB percent: 38.2%

TWO ZONE

16/06/2021 15:22:09 - INFO - activitysim.core.mem - trace_memory_info tour_mode_choice.work.simple_simulate.chunk_1.eval_nl.eval_utils.add.expression_values rss: 62.72GB used: 137.57 GB percent: 43.0%
16/06/2021 15:24:51 - INFO - activitysim.core.mem - trace_memory_info tour_mode_choice.work.simple_simulate.chunk_1.eval_nl.eval_utils.add.utilities rss: 62.83GB used: 149.6 GB percent: 46.8%

2 - the expression files are very similar

ONE ZONE

https://github.com/esanchez01/activitysim/blob/develop/activitysim/examples/example_mtc/configs/tour_mode_choice.csv

TWO ZONE

https://github.com/esanchez01/activitysim/blob/develop/activitysim/examples/example_psrc/configs/tour_mode_choice.csv

3 - Some more specific logging within a process shows roughly the same number of tours being processed in the chunk and similar tour mode shares but longer runtimes

ONE ZONE

Line 2703 in mp_households_0-activitysim.log

The chunk runtime is 6 seconds for 7029 tours and the mode share is reported

DRIVEALONEFREE 2044
SHARED2FREE 1673
SHARED3FREE 1528
WALK 830
WALK_LRF 298
WALK_EXP 164
WALK_LOC 130
TNC_SINGLE 120
BIKE 80
TAXI 44

TWO ZONE

Line 2678 in mp_households_0-activitysim.log

The chunk runtime is 166 sec for 7087 tours and the mode share is basically the same

DRIVEALONEFREE 2028
SHARED2FREE 1705
SHARED3FREE 1560
WALK 897
WALK_LRF 267
WALK_EXP 159
WALK_LOC 115
TNC_SINGLE 110
BIKE 94
TAXI 33

4 - so it seems we need to debug/profile a bit to figure out exactly why.

The network_los engine/setup, and potentially the MAZ to MAZ costs are what's different so it's likely something in there. Without actually profiling, we're wondering about the bellow note in the quick_loc_df() called by get_mazpairs()....

• https://github.com/esanchez01/activitysim/blob/develop/activitysim/examples/example_psrc/configs/network_los.yaml#L16
• expressions such as od_skims['DISTWALK']

• activitysim/activitysim/core/util.py

  Line 233 in a57efa1

  pandas DataFrame.loc[] indexing doesn't scale for large arrays (e.g. > 1,000,000 elements)

• activitysim/activitysim/core/los.py

  Line 517 in a57efa1

  s = util.quick_loc_df(i, self.maz_to_maz_df, attribute)

[bstabler]

We're going to investigate some more to confirm the hypothesis and to hopefully develop an improvement. We'll look at the DaySim and CTRAMP code for ideas as well. Testing with the full scale setup will be important. This would benefit both 2 and 3 zone setups.

[bstabler]

@esanchez01- I'm in the process of testing this improvement on a full scale version of your model and hoping you can test it as well?

[esanchez01]

@bstabler, I manually made the change you mentioned to v0.9.9.1, as this is the version my previous tests ran on (no chunking update), and reran the full SANDAG 2-Zone PSRC-based example. There are some noticeable run time improvements when compared to the equivalent 2-Zone run I shared the log files for:

model_name	Pre-Update	Post-Update	Pre - Post
compute_accessibility	16.4	15.4	1
school_location	15.8	10.8	5
workplace_location	30.7	23.6	7.1
mandatory_tour_scheduling	27.4	13.5	13.9
joint_tour_destination	12	1.7	10.3
non_mandatory_tour_destination	40.1	26.5	13.6
tour_mode_choice_simulate	25.9	18.5	7.4
atwork_subtour_destination	6.3	4	2.3
trip_destination	153.9	48.9	105
trip_purpose_and_destination	7.6	4.6	3
trip_mode_choice	21.1	17.5	3.6
write_trip_matrices	8	13.4	-5.4
SUB MODEL RUN TIME	388.1	220.8	167.3
TOTAL RUN TIME	442	258	184

I will rerun the 3-Zone example with this change as well.

[bstabler]

I like these improvements. Tour mode choice is still quite a bit slower than the 1 zone example, which, if I recall, was a couple minutes, so I'll keep investigating. @esanchez01 - can you post the 1 zone example runtimes by submodel for reference?

[esanchez01]

Here are the results of the 3-Zone rerun:

model_name	Pre-Update	Post-Update	Pre - Post
compute_accessibility	42.3	39.3	3
school_location	21.4	13.3	8.1
workplace_location	41.1	28.2	12.9
mandatory_tour_scheduling	31.8	17.1	14.7
joint_tour_destination	17.5	4.5	13
non_mandatory_tour_destination	49.3	31.7	17.6
tour_mode_choice_simulate	38.5	30.3	8.2
atwork_subtour_destination	7.5	5.1	2.4
atwork_subtour_mode_choice	4.7	3.6	1.1
trip_destination	135.2	67.5	67.7
trip_purpose_and_destination	7.8	5.2	2.6
trip_mode_choice	19	15.7	3.3
SUB MODEL RUN TIME	442	288.5	153.5
TOTAL RUN TIME	489	328	161

[esanchez01]

I like these improvements. Tour mode choice is still quite a bit slower than the 1 zone example, which, if I recall, was a couple minutes, so I'll keep investigating. @esanchez01 - can you post the 1 zone example runtimes by submodel for reference?

Complete run time list for an optimized 1-Zone run:

model_step	1-Zone
initialize_landuse	0
initialize_households	1.3
compute_accessibility	0.9
school_location	4.8
workplace_location	10.9
auto_ownership_simulate	0
free_parking	0.1
cdap_simulate	0.4
mandatory_tour_frequency	0.1
mandatory_tour_scheduling	6.1
joint_tour_frequency	0.1
joint_tour_composition	0
joint_tour_participation	0.2
joint_tour_destination	1
joint_tour_scheduling	0.2
non_mandatory_tour_frequency	3
non_mandatory_tour_destination	10
non_mandatory_tour_scheduling	2.6
tour_mode_choice_simulate	1.2
atwork_subtour_frequency	0.1
atwork_subtour_destination	1.3
atwork_subtour_scheduling	0.3
atwork_subtour_mode_choice	0.1
stop_frequency	0.4
trip_purpose	0
trip_destination	27.1
trip_purpose_and_destination	0
trip_scheduling	8.3
trip_mode_choice	1.2
write_data_dictionary	0.2
write_trip_matrices	6.6
write_tables	3.3
SUB MODEL RUN TIME	91.8
TOTAL RUN TIME	112