'utility not aligned' error when running sharrow in test mode using uv, but not in conda using same version of Activitysim.

[stefancoe]

Describe the bug
This is an odd one. When running sharrow in test mode with a uv environment, I get a 'utility not aligned' error here, but not when running the same version of activitysim using anaconda. I did some debugging and the utilities that are causing the issue are coming from the bike and walk tour mode choice utilities when getting logsums during mandatory_tour_scheduling step, so perhaps this has something to do with the maz_to_maz skims. When using the anaconda env, the utilities from variables utilities and sh_util are aligned. Interestingly, when switching to an older version of Activitysim the bug turned up sooner in the disaggregate accessibilities step. I tested this against an older anaconda environment and a new one. For the new one, I first created the environment and then ran "pip install -e" from the (cloned) activitysim directory. This essentially installed activitysim in editable mode and then all of its dependencies using pip.

To Reproduce
Steps to reproduce the behavior:

1. Copy conifgs and input data from here.
2. Clone Activitysim
3. Open a command prompt, cd to Activitysim, then enter uv sync --locked to create the uv .venv.
4. Enter .venv\Scripts\activate to activate the uv environment.
5. Enter activitysim run -c path_to_configs_sh_compile -c path_to_configs_dev -d path_to_data_test -o path_to_output
6. The model should crash during mandatory tour scheduling.
7. Create a new anaconda env by using the environment.yml file in the google drive linked above or by creating a new env and then running pip install -e in the activitysim root dir.
8. Run the model using the anaconda env using the same arguments as above.
9. The model should not crash.

Additional context
This is probably a library version issue and not anaconda/uv issue. Either way it will be good to know why the utilities are different running one way and not the other.

[i-am-sijia]

@stefancoe I confirm I was able to replicate steps 1-6, and got the utility not aligned error in mandatory tour scheduling.

I haven't been able to replicate step 7, though. The environment.yaml you shared installs dependencies using the default Anaconda distribution channel, which gives me access issue because WSP does not have license to Anaconda. It still doesn't work after I change the channel to nodefaults and conda-forge because it could not find some dependency versions on conda-forge. I do notice that you are using an older activitysim==1.3.0 and the latest sharrow==2.15.0 which seem to work in this case.

To create a conda install, I went back to the old activitysim-dev-base.yaml from v1.3.4, using that to create an ActivitySim conda env. It's currently stuck in the conda env creation step for the past hour (Executing transactions...), I will wait a little longer to let it complete and then try testing step 8.

[stefancoe]

Thanks @i-am-sijia. I think the issue is with the sharrow cache folder/files. I deleted it, re-ran using uv and the model finished without crashing. Do you have a sharrow cache on your machine? If so, can you try deleting and then re-running with uv?

Sorry about the environment.yaml file having the wrong version of activitysim- I exported that when I was testing an earlier version.

[stefancoe]

@jpn-- The issue we are running into now is that the model crashes during trip destination when compiling sharrow with skims that have gaps in the zone system. It also crashes in trip destination when pointed to a compiled sharrow cache made from our smaller dataset. The model does not crash when we limit the skims to internal zones only. We use the full skims for other models (trucks, supplementals) so it would be ideal to only use this one set of OMX skims (with the gaps) throughout Soundcast. You can find all the data here. This includes all the configs folders, a sharrow_cache folder, and a data dir that has three folders: all_zones, internal_zones & data_test_new_skims, which is the small dataset we use to compile sharrow. Model is run like so:

Sharrow compile:
activitysim run -c configs_sh_compile -c configs_dev_NEW_SKIMS -o output_dir

Production mode with sharrow:
activitysim run -c configs_sh -c configs_mp -c configs_dev_NEW_SKIMS -o output_dir

[jpn--]

I have narrowed this problem down to what I will call a "corner case", although I don't know how prevalent it would actually be in practice. In short, the problem is in the handling of the "DataFrameMatrix" class, which is an oddball legacy object that is only ever used in trip destination. This class is not compatible with sharrow, which was previously known; it is the one "special case" thing where sharrow and legacy spec files actually need to have two different expressions in the trip destination spec file. This special case handling is failing when there are TAZ codes that correspond to external zones (or for some other reasons are TAZs that have no MAZs associated with them) AND when these TAZ-with-no-MAZ zones have numbers that are in the middle of the run of TAZs (i.e. there are regular internal zones with numbers on both sides). I believe I have a fix put together but I am setting up a proper small test dataset (not the full-scale PSRC data I've been debugging) to reproduce the error and then confirm that the fix works.

[jpn--]

@stefancoe I opened a PR that should address this issue. Can you review, and confirm that it solves your problem?

[stefancoe]

@jpn-- I should be able to check in the next day or so.

[stefancoe]

@jpn-- Sorry for the delay. The model runs to completion when running without sharrow, but still crashes during trip destination when using sharrow. The log is attached. Let me know if there is anything I can do on my end.

activitysim.log

[jpn--]

For those following this (esp @stefancoe ), a short update: I have identified a bug that is likely the cause of this problem. Unfortunately, it is a bug not in our code, but in a dependency (the sparse library), and it is one of those "corner cases": there is an overflow in indexing into sparse (MAZ-to-MAZ) LOS matrix data, which occurs only on Windows and only when the number of MAZs exceeds 46,340 (i.e. sqrt(2^31)).

We have seen and solved similar bugs before, but they were in our code, while this is in a dependency. We could try to push a fix to that library, but there is a better, more robust solution: un-pin our dependency on numpy<1.26 and move up to numpy 2.x. This will solve the bug at hand, as well as prevent other related issues, as numpy 2.x enforces a default integer of int64 on all platforms (including Windows).

There are numerous other small cascading changes that need to be made with this update, which I am working on, as various other dependencies also will need to be updated because of this change, including pandera for input checking, and maybe others. Hopefully we will have a complete fix soon.

[jpn--]

@stefancoe can you see if the changes now pushed to #1027 address the crash you observed? Note you will need to make sure the UV environment updates with the updated dependencies (check that numpy is version 2.x) and do run with your full zone system again. Thanks!