would be helpful to have an easy way to keep information about every mutation generated

[bhaller]

This came up during my multitrait testing; for a model with seven traits and three mutation types, I wanted to compare the distribution of effects and dominance coefficients, as generated by SLiM versus at the end of the simulation, for each trait/muttype. You can do this, with a mutation() callback, but for so many combinations, it becomes quite a pain. It would be nice to have a facility to automatically log this kind of information.

So I just did it, and it seems to work nicely. I'm making a new issue about it in case anybody has any comments/requests regarding this facility. Maybe @petrelharp @vsudbrack @jiseonmin @DrK-Lo might be interested. Note that I'm perfectly happy with the design as it stands now; don't feel a need to suggest changes just for the sake of saying something, "LGTM" is an entirely acceptable response. :-> But if I've dropped the ball here in some way, with respect to what people would like to be able to do with this sort of facility, please do let me know!

OK, so here is the doc for it:

– (fo<DataFrame>)loggedData(string$ kind, [logical$ id = F], [logical$ mutationTypeID = F], [logical$ chromosomeID = F], [logical$ position = F], [logical$ nucleotideValue = F], [logical$ originTick = F], [logical$ subpopID = F], [logical$ tag = F], [Niso<Trait> trait = NULL], [logical$ effect = F], [logical$ dominance = F], [logical$ hemizygousDominance = F])
Returns mutation data produced by the mutation type’s logging facility, as configured by logMutationData(). The data returned can be in the form of means across all logged mutations (for kind="mean"), standard deviations across all logged mutations (for kind="sd"), or separate values for each mutation (for kind="values"). If logging only of means was enabled (with the meanOnly=T option to logMutationData()), only kind="mean" is allowed, since separate values for each mutation are then not logged.
The remaining flags control which columns of data should be returned; see logMutationData() for a summary of the mutation properties they refer to. If only one data column is specified by the flags, a vector of values (or a mean or standard deviation) will be returned for that one specified data column. If more than one flag is set to T, a DataFrame object will be returned with named columns of values (or means, or standard deviations) for each specified data column. The trait property specifies which traits values should be returned for, with respect to the effect, dominance, and hemizygousDominance flags; see logMutationData() for further description. If all of these flags are F (the default), that is taken to mean that all logged data should be returned; in that case, a vector will be returned if only one column of data was logged, otherwise a DataFrame object will be returned, the same as when flags are specified explicitly. Flags set to T for data columns that were not actually logged will simply be ignored; similarly, traits specified by trait that were not actually logged will simply be ignored. See the Eidos manual for the DataFrame class documentation.

– (void)logMutationData(logical$ enable, [logical$ autogeneratedOnly = T], [logical$ meanOnly = F], [logical$ id = F], [logical$ mutationTypeID = F], [logical$ chromosomeID = F], [logical$ position = F], [logical$ nucleotideValue = F], [logical$ originTick = F], [logical$ subpopID = F], [logical$ tag = F], [Niso<Trait> trait = NULL], [logical$ effect = F], [logical$ dominance = F], [logical$ hemizygousDominance = F])
Starts or ends logging of data about new mutations belonging to the target mutation type. If autogeneratedOnly is T (the default), only new mutations generated automatically by SLiM will be logged (including mutations that are substituted in for an auto-generated mutation using a mutation() callback; that is still considered part of the auto-generation process). If autogeneratedOnly is F, mutations generated in script, such as with addNewMutation(), addNewDrawnMutation(), and reading from files such as VCF, MS, or .trees, will also be logged. The logged information can be obtained later with the loggedData() method. Once logging has been started with enable=T it cannot be modified, only stopped with enable=F; and if logging is subsequently resumed with enable=T, any previously logged data will be discarded. (This can be useful if you wish to limit the size of the in-memory data while continuing to log new data: periodically write the accumulated data to a file and then disable and re-enable logging to discard the old data.)
If meanOnly is F (the default), values for each new mutation will be kept separately. Beware: the memory usage entailed by this option can be extremely large! Alternatively, if meanOnly is T, only a running sum, used to compute a mean, will be kept for each type of data; the memory usage for this option will be small and constant, but of course a mean is more useful for some columns of data than others. If per-mutation data is desired for any one column, use meanOnly=F; this option cannot be controlled independently for the various columns of data being logged.
Next are parameters that control which mutation properties will be logged: id controls the id property; mutationTypeID controls the id property of the mutation’s mutation type (this will be the same for all mutations logged by a given MutationType, it can be useful if you combine datasets from more than one mutation type later); chromosomeID controls the id property of the mutation’s associated chromosome; position controls the position property; nucleotideValue controls the nucleotideValue property; originTick controls the originTick property; subpopID controls the subpopID property; and tag controls the tag property. Data columns will be added in the order of these parameters; the id column will be first, if requested, for example.
Last come parameters that control the logging of trait-associated data for the new mutations. The trait parameter controls which traits will be logged. The traits can be specified as integer indices or string names of traits in the species, or directly as Trait objects; NULL represents all of the traits in the species. For each specified trait, the effect parameter controls logging of the effect size, the dominance parameter controls the dominance, and the hemizygousDominance parameter controls the hemizygous dominance. Data columns for this trait-associated data will be grouped by trait; for example, if two traits named height and weight are specified, and the effect=T and dominance=T flags are specified, then columns heightEffect, heightDominance, weightEffect, and weightDominance will be added, in that order.
Note that logging occurs after all mutation() callbacks have been called, at the point when the new mutation is actually added to the simulation. If addition of a new mutation is prevented, by a mutation() callback or by the current stacking policy, that mutation will not be logged. The information logged will be the mutation’s properties at the moment that it is added; a tag value set by a mutation() callback will therefore be captured in the logged data, for example. Changes to mutations made after that point will not be preserved in the log; the log is a snapshot of the moment of each mutation’s addition to the simulation.

This has been committed to the multitrait branch, which seems to be working nicely so far actually, although a great deal of work remains. :-> Documentation for multitrait beyond what is provided in SLiMgui is not ready to share, though.

[vsudbrack]

Hey Ben!

That's nice. I was thinking that perhaps another summary statistics that could be worth adding is the total number of mutations created so far, like a column 'n'. I guess it would be equivalent to having kind='values' and then getting the length of the output vector, but imagine for kind='mean' that information would have been lost, and sometimes it might be helpful to know how many values are being considered within the mean.

[jiseonmin]

Hi Ben,
Thank you for the update! Actually I've been using a script where I manually write the effect sizes of mutations to a csv file for my project, so this will be really useful for me! Currently, I let a vector of mutational information grow until the very end of simulation and then export it to csv, but it makes sense you want to refresh the in-memory data by writing the current data out periodically. And in my case, I will need to specify on what tick the mutation data was collected, so it may be useful to have that as an option for another column? (although it's probably not hard to record it separately) Also maybe it will be useful to have an option to record mutations for sampled individuals instead of an entire subpopulation so that one can save even more memory? This seems doable with tag, but I wonder if there is a more efficient way.

[bhaller]

Hey Ben!

Hi Vitor!

That's nice. I was thinking that perhaps another summary statistics that could be worth adding is the total number of mutations created so far, like a column 'n'. I guess it would be equivalent to having kind='values' and then getting the length of the output vector, but imagine for kind='mean' that information would have been lost, and sometimes it might be helpful to know how many values are being considered within the mean.

I'll add a kind="count" option to loggedData(), good idea.

[bhaller]

Hi Ben,

Hi Jiseon!

Thank you for the update! Actually I've been using a script where I manually write the effect sizes of mutations to a csv file for my project, so this will be really useful for me! Currently, I let a vector of mutational information grow until the very end of simulation and then export it to csv, but it makes sense you want to refresh the in-memory data by writing the current data out periodically.

OK, glad it will be useful! I suspected that it might be, although nobody has ever asked for it. :->

And in my case, I will need to specify on what tick the mutation data was collected, so it may be useful to have that as an option for another column? (although it's probably not hard to record it separately)

The originTick of the mutation can already be logged; or do you mean something else?

Also maybe it will be useful to have an option to record mutations for sampled individuals instead of an entire subpopulation so that one can save even more memory? This seems doable with tag, but I wonder if there is a more efficient way.

Hmm, interesting. Since individuals are constantly coming and going, I'm not sure how one would efficiently tell SLiM "gather information for mutations in this individual but not this individual"; and there's a bit of trickiness there, too, since mutations happen during gametogenesis, so they are kind of in-between the parent individual and the child individual, and you might want to filter based on either/both, so it gets conceptually complicated I guess. I think maybe using tag would be the best way to go on this, yes – you could tag new mutations, in a mutation() callback, based on whatever criterion you wanted, and then apply that tag-based filter to the dataset at the end. If you're flushing the data out to disk periodically anyway, it shouldn't be too painful, I think? Memory is expensive, but disk is cheap. :->

But I can certainly imagine an option for logMutationData() that says "only record mutations with a tag value of x", and it wouldn't be hard to implement. If you think that is compelling enough to be worth the additional API complexity, let me know and I'll be happy to add it! Probably something like [Ni$ filterByTag = NULL], where NULLmeans "don't filter" and any otherinteger` value means "only record mutations with the given tag value"...

[DrK-Lo]

Thanks Ben! Just as a side note I found this confusing: "If all of these flags are F (the default), that is taken to mean that all logged data should be returned;" K. E. Lotterhos, PhD (she/her) Department of Marine and Environmental Sciences Coastal Sustainability Institute Northeastern University Marine Science Center Nahant, MA 01908 I reply to emails mid-day on weekdays

…

________________________________ From: Ben Haller ***@***.***> Sent: 12 January 2026 10:12 To: MesserLab/SLiM ***@***.***> Cc: Lotterhos, Katie ***@***.***>; Mention ***@***.***> Subject: Re: [MesserLab/SLiM] would be helpful to have an easy way to keep information about every mutation generated (Issue #607) [https://avatars.githubusercontent.com/u/10171101?s=20&v=4]bhaller left a comment (MesserLab/SLiM#607)<#607 (comment)> Hi Ben, Hi Jiseon! Thank you for the update! Actually I've been using a script where I manually write the effect sizes of mutations to a csv file for my project, so this will be really useful for me! Currently, I let a vector of mutational information grow until the very end of simulation and then export it to csv, but it makes sense you want to refresh the in-memory data by writing the current data out periodically. OK, glad it will be useful! I suspected that it might be, although nobody has ever asked for it. :-> And in my case, I will need to specify on what tick the mutation data was collected, so it may be useful to have that as an option for another column? (although it's probably not hard to record it separately) The originTick of the mutation can already be logged; or do you mean something else? Also maybe it will be useful to have an option to record mutations for sampled individuals instead of an entire subpopulation so that one can save even more memory? This seems doable with tag, but I wonder if there is a more efficient way. Hmm, interesting. Since individuals are constantly coming and going, I'm not sure how one would efficiently tell SLiM "gather information for mutations in this individual but not this individual"; and there's a bit of trickiness there, too, since mutations happen during gametogenesis, so they are kind of in-between the parent individual and the child individual, and you might want to filter based on either/both, so it gets conceptually complicated I guess. I think maybe using tag would be the best way to go on this, yes – you could tag new mutations, in a mutation() callback, based on whatever criterion you wanted, and then apply that tag-based filter to the dataset at the end. If you're flushing the data out to disk periodically anyway, it shouldn't be too painful, I think? Memory is expensive, but disk is cheap. :-> But I can also imagine an option for logMutationData() that says "only record mutations with a tag value of x", and it wouldn't be hard to implement. If you think that is compelling enough to be worth the additional API complexity, let me know and I'll be happy to add it! — Reply to this email directly, view it on GitHub<#607 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABUNI3NL6OORHEA3YE734B34GO2UTAVCNFSM6AAAAACRLT6E5CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTOMZZGA2DOMRRGA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[bhaller]

Thanks Ben!

Hi Katie!

Just as a side note I found this confusing:
"If all of these flags are F (the default), that is taken to mean that all logged data should be returned;"

Yes, it's a weird design choice, but I am prepared to defend it. :-> The thing is, people will want to use this functionality in one of two ways, I think: (a) record EVERYTHING, or (b) record just one or a couple of things. To make (b) easy, all the flags need to default to F; then you can just call logMutationData() with, say, effect=T to log mutation effects, and not have to explicitly pass F for all of the other flags. So, OK, all the flags need to default to F. But then how do you easily do (a), record EVERYTHING? Making the user pass a=T, b=T, c=T, d=T, e=T, f=T... to turn everything on is just super annoying, and is also a bit fragile since if I add new flags later on, the user would have to notice that and add x=T, y=T, z=T to their list of options. But having all of the flags F, the default state, is not useful in itself – it would just say "turn recording on, but record NOTHING", which nobody would ever do in practice. So, why not use that state as a special shorthand for (a), record EVERYTHING? So yes, I realize it is weird. :-> But I think when people actually start using this method they will discover that it is actually a nice convenience. (Or maybe I'm wrong and they will continue to just think it's weird, lol. It's hard to guess!)

[DrK-Lo]

Why not make the default all flags set to TRUE? K. E. Lotterhos, PhD (she/her) Department of Marine and Environmental Sciences Coastal Sustainability Institute Northeastern University Marine Science Center Nahant, MA 01908 I reply to emails mid-day on weekdays

…

________________________________ From: Ben Haller ***@***.***> Sent: 12 January 2026 10:22 To: MesserLab/SLiM ***@***.***> Cc: Lotterhos, Katie ***@***.***>; Mention ***@***.***> Subject: Re: [MesserLab/SLiM] would be helpful to have an easy way to keep information about every mutation generated (Issue #607) [https://avatars.githubusercontent.com/u/10171101?s=20&v=4]bhaller left a comment (MesserLab/SLiM#607)<#607 (comment)> Thanks Ben! Hi Katie! Just as a side note I found this confusing: "If all of these flags are F (the default), that is taken to mean that all logged data should be returned;" Yes, it's a weird design choice, but I am prepared to defend it. :-> The thing is, people will want to use this functionality in one of two ways, I think: (a) record EVERYTHING, or (b) record just one or a couple of things. To make (b) easy, all the flags need to default to F; then you can just call logMutationData() with, say, effect=T to log mutation effects, and not have to explicitly pass F for all of the other flags. So, OK, all the flags need to default to F. But then how do you easily do (a), record EVERYTHING? Making the user pass a=T, b=T, c=T, d=T, e=T, f=T... to turn everything on is just super annoying, and is also a bit fragile since if I add new flags later on, the user would have to notice that and add x=T, y=T, z=T to their list of options. But having all of the flags F, the default state, is not useful in itself – it would just say "turn recording on, but record NOTHING", which nobody would ever do in practice. So, why not use that state as a special shorthand for (a), record EVERYTHING? So yes, I realize it is weird. :-> But I think when people actually start using this method they will discover that it is actually a nice convenience. (Or maybe I'm wrong and they will continue to just think it's weird, lol. It's hard to guess!) — Reply to this email directly, view it on GitHub<#607 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABUNI3LVWTRY26ER32B62U34GO327AVCNFSM6AAAAACRLT6E5CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTOMZZGEYDOMBZG4>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[bhaller]

Why not make the default all flags set to TRUE?

Then (b), wanting to turn on just one or a few of the flags, would become extremely annoying; you'd have to pass a=T for the option you want, and then b=F, c=F, d=F, e=F, f=F, g=F.... So the default for all flags really needs to be F.

[bhaller]

Just as a side note I found this confusing:
"If all of these flags are F (the default), that is taken to mean that all logged data should be returned;"

Yes, it's a weird design choice, but I am prepared to defend it. :-> The thing is, people will want to use this functionality in one of two ways, I think: (a) record EVERYTHING, or (b) record just one or a couple of things. To make (b) easy, all the flags need to default to F; then you can just call logMutationData() with, say, effect=T to log mutation effects, and not have to explicitly pass F for all of the other flags. So, OK, all the flags need to default to F. But then how do you easily do (a), record EVERYTHING? Making the user pass a=T, b=T, c=T, d=T, e=T, f=T... to turn everything on is just super annoying, and is also a bit fragile since if I add new flags later on, the user would have to notice that and add x=T, y=T, z=T to their list of options. But having all of the flags F, the default state, is not useful in itself – it would just say "turn recording on, but record NOTHING", which nobody would ever do in practice. So, why not use that state as a special shorthand for (a), record EVERYTHING? So yes, I realize it is weird. :-> But I think when people actually start using this method they will discover that it is actually a nice convenience. (Or maybe I'm wrong and they will continue to just think it's weird, lol. It's hard to guess!)

Playing around with this further, though, I realized that it just isn't a good idea. Apart from being weird and confusing (as evidenced by this thread), I realized that it also doesn't really work as a solution to the problem, because there are two flags that the user will probably not want turned on, most of the time. The nucleotideValue flag will end up producing an error if the model is not nucleotide-based; and the tag flag will end up producing a "used before set" error for the mutation tag values unless the user does happen to be setting mutation tag values in their model, which is rare. So setting those flags to T automatically, when all flags are F, doesn't make sense. At that point the idea becomes too bogged down in minutiae to be salvageable. So, never mind. :-> If you want a bunch of flags to be turned on, you will have to explicitly turn on all of those flags; no shortcut. Such is life. API design is hard, lol.

[bhaller]

(If anybody cares, I just pushed a couple of fixes to multitrait for this, added the kind="count" feature, and removed the "if everything is F than everything is T" API weirdness, as discussed above. AFAICT the feature is ready to use, but the multitrait branch is not a particularly stable/tested place right now. :->)

[petrelharp]

This does look very useful! Next, there will be requests for this sort of thing for individuals (to record spatial location, phenotype, tag value, etcetera each tick).

My only suggestion is to do away with the "if one then then returns a vector otherwise a data frame" wierdness - I see where you're coming from, but there's good reasons why it is largely regarded as a bad move that certain base R functions (e.g., "[") do this same thing. For instance, first I write my code to just log one thing. Then I realize that I need to log another thing - whoops, this breaks a bunch more of my recipe than if the return type was consistent. And any code that programatically decides whether to record one or two columns is more complex.

[bhaller]

This does look very useful! Next, there will be requests for this sort of thing for individuals (to record spatial location, phenotype, tag value, etcetera each tick).

Yes, this occurred to me. But that's OK – if it's useful, I don't mind. :-> But I'm not sure it's as useful, really. In general, logging stuff and writing it to a file is not a big deal. What drove me to add this feature was the multiplicity of the data – with seven traits and three mutation types in the test model I was working on, with effect size and dominance coefficient for each one, that was 42 things to set up logs for if I wanted to be able to analyze it all, and that's clearly just not reasonable without help from SLiM. If there are other situations where simple logging code doesn't cut the mustard, I'm happy to oblige. :->

My only suggestion is to do away with the "if one then then returns a vector otherwise a data frame" wierdness - I see where you're coming from, but there's good reasons why it is largely regarded as a bad move that certain base R functions (e.g., "[") do this same thing. For instance, first I write my code to just log one thing. Then I realize that I need to log another thing - whoops, this breaks a bunch more of my recipe than if the return type was consistent. And any code that programatically decides whether to record one or two columns is more complex.

Interesting. So... you want it to make a DataFrame even when the user asks for just a single piece of data? I can see your reasoning, but... just as often, if not more so, there will be cases where the user really just wants to get one thing, and getting a DataFrame in response to that simple query will be an annoyance. I suppose I could add a parameter to control whether they get a vector or a DataFrame, but that's just that much more complexity in the API and the code. So. I'll do what you advise, since you've got a lot more experience with R (not to mention Python) than I do, but be careful what you wish for. :-> What's your final call? Always a DataFrame?

[petrelharp]

What's your final call? Always a DataFrame?

What you say makes lots of sense... and, that's my vote - always a DataFrame. The annoyances the other way are harder to get around.