Idents?

[mdhaney]

I'm so excited that you added lookup refs and upsert support! Trying them out right now.

I noticed in the changelog you also added entid support - so does that mean I can now specify :db/ident for entities?

[sparkofreason]

+1. Lookup refs definitely a big improvement when working with data from Datomic, but :db/ident support for ref attributes would be extra tasty.

[tonsky]

Well, idents are trickier because they not only affect input (you specify keyword, it gets resolved to eid), but also output (e.g. in Datomic results you won’t get entity id or attribute id if that eid has ident, you’ll get keyword in instead).

Datomic sure needs idents to save bandwidth/storage (so they can store integer attribute ids instead of strings/keywords which take a lot of space). They explicitly don’t recommend using idents for anything else but attrs and enum values (see http://docs.datomic.com/identity.html#sec-3).

But for in-memory implementation like DataScript you don’t need to save space (you operate with pointers anyway, so all keywords point to the same location in memory). You can also use keywords directly to name attributes and enum values. I find this model easier to understand and easier too implement too.

Idents are also not free, they have to be resolved twice (on input and on output). That’s extra hashmap or index lookup per, say, returned entity.

So I’m still deciding if idents are needed. If you guys can name couple of use-cases, it would be helpful.

Meanwhile, ident resolution can be emulated with [:db/ident :special-key] kind of lookup refs.

[sparkofreason]

Our main use-case is the ability to share query definitions between Datomic and Datascript. We're pushing the "Datomic-as-protocol" idea as far as we can, and it's working out very well, but API differences like this add friction. For instance, we're able to develop disconnected from Datomic using Datascript along with some CLJX. This makes for rapid client development, but then we want to hook up to a server and use the same queries built in the disconnected case.

[tonsky]

I understand that. My question is how are you using idents in Datomic then.

[sparkofreason]

We have various type "enums" which we mix and match, mainly as filters for list views. The types are also used as metadata to specify which attributes go with which entity types, for the purpose of auto-building editor forms, etc.

[mdhaney]

My use case is that I create an "app" entity for various housekeeping stuff, and it's inconvenient to have to store the entity id to refer to after creating this entity. I've been cheating by using 0 for the eid, but that felt kludgy to me.

Idents would work here, but lookup refs should work just as well, I.e. I can add something like :entity.type :app and use that as a lookup ref to easily refer to it.

So yeah, if there are performance implications, I would rather not have you pursue it.

[sparkofreason]

For my case, I think it would suffice to handle only the input side of things. That avoids the necessity to pull from Datomic all of the enum entities in addition to the data which references them.

[sparkofreason]

I've found away to work around this and keep things consistent on the Datomic/Datascript sides, so I'll retract my request for this feature.

[cigitia]

I'd like to affirm that I find idents useful in both use cases mentioned by others above:

• I use idents to easily use singleton objects, such as an “app” entity that contains things like whether a certain UI mode is on or off. If one needed to store the text currently entered into a box an application’s UI, there is then no other place to store it other than some single entity—and that single entity will need to be modified in transactions in the future.
• I also use idents to refer easily to enumerated objects, such as states or categories of other entities, in code that needs to refer to specific categories. Those categories might also have attributes of their own, such as end-user display names, but each one changes the application's logic in a different way, so the code need to be able to directly refer to them, especially in transactions.

Without idents, as far as I know, in order to modify any of those singleton and singleton entities in transactions, their IDs must be stored beforehand separately from the database, either manually hardcoded into code using fragile magic numbers—or manually retrieved sometime after the database is initialized with their them. Those IDs must then be manually passed into every transaction that involves them. Storing all of these ID constants and using them in transactions aren’t a big deal when there is only one singleton object, but they become an issue when there are many singleton or enumerable objects for which to store IDs, and lookup keys don't really help with this.

The Entity and Pull APIs also would benefit from idents, giving the ability to easily retrieve all attributes of singleton entities at once, without having to manually store their IDs elsewhere—I use this while serializing a singleton object's state into a non-EDN text format.

Idents would be very useful.

Thanks again for DataScript, though; I'm thankful for how amazing it already is.

[tonsky]

Ok, I’ll look into it later

On Sat, Mar 7, 2015 at 11:01 AM cigitia notifications@github.com wrote:

I'd like to affirm that I find idents useful in both use cases mentioned
by others above:

• I use idents to easily use singleton objects, such as an “app”
  entity that contains things like whether a certain UI mode is on or off. If
  one needed to store the text currently entered into a box an application’s
  UI, there is then no other place to store it other than some single
  entity—and that single entity will need to be modified in transactions in
  the future.
• I also use idents to refer easily to enumerated objects, such as
  states or categories of other entities, in code that needs to refer to
  specific categories. Those categories might also have attributes of their
  own, such as end-user display names, but each one changes the application's
  logic in a different way, so the code need to be able to directly refer to
  them, especially in transactions.

Without idents, as far as I know, in order to modify any of those
singleton and singleton entities in transactions, their IDs must be stored
beforehand separately from the database, either manually hardcoded into
code using fragile magic numbers
https://en.wikipedia.org/wiki/Magic_number_(programming)#Unnamed_numerical_constants—or
manually retrieved sometime after the database is initialized with their
them. Those IDs must then be manually passed into every transaction that
involves them. Storing all of these ID constants and using them in
transactions aren’t a big deal when there is only one singleton object, but
they become an issue when there are many singleton or enumerable objects
for which to store IDs, and lookup keys don't really help with this.

The Entity and Pull APIs also would benefit from idents, giving the
ability to easily retrieve all attributes of singleton entities at once,
without having to manually store their IDs elsewhere—I use this while
serializing a singleton object's state into a non-EDN text format.

Idents would be very useful.

Thanks again for DataScript, though; I'm thankful for how amazing it
already is.

—
Reply to this email directly or view it on GitHub
#52 (comment).

[sparkofreason]

I've been able to work around some of the Datomic/Datascript differences with a few small utility functions. See gist: https://gist.github.com/sparkofreason/6b3ffd63d148cd7dc37a

[tonsky]

Yep, looks painful :) I’ll try to add idents to DataScript soon

(extend-type UUID
IComparable
(-compare [x y](compare %28datascript/squuid-time-millis x%29
%28datascript/squuid-time-millis y%29)))

this should probably go as a patch to CLJS itself. Only compare UUIDs
as strings, without datascript semantics

On Sun, Mar 8, 2015 at 12:17 PM Dave Dixon notifications@github.com wrote:

I've been able to work around some of the Datomic/Datascript differences
with a few small utility functions. See gist:
https://gist.github.com/sparkofreason/6b3ffd63d148cd7dc37a

—
Reply to this email directly or view it on GitHub
#52 (comment).

[sparkofreason]

Not that painful, really. If it were a choice between using the utility functions or having idents but degraded Datascript perf, I'd take the first option. Personally I think the problem is really with Datomic. The enum abstraction is leaky when used through the pull API, would rather see it fixed there.

[cigitia]

Idents are still important for managing many singleton and enumerated entities, particularly in transactions and query/entity/pull inputs. That Gist above would help out with resolving idents in pull-API entities, but it does seem painful, and it only works for the pull API anyway, not transactions, where I find the lack of idents the most painful.

If output performance still is that large of a concern, it might be mitigated in a couple of ways:

• By supporting idents in transactions and query/pull inputs but not resolving them in query/entity/pull outputs. Support for idents in output matters less, at least for me, since I would be able to predict wherever an ident would be expected to appear in an output's datoms and do a simple check myself later.
• Or by adding ident resolution as a conditional option in the API's functions, which would enable only people who are willing to pay for ident resolution to get it.
• Or by providing utility functions similar to that of the Gist above that would transform query inputs, pull inputs, query outputs, entities, etc., replacing idents within them with their IDs, or vice versa.
• Or something else I haven't thought up of.

I think I prefer the first choice the most. But in the end, as always, it's difficult to predict just how much ident resolution would affect performance in general anyway without actually trying it. It might even end up not being a significant problem at all.

But either way, idents remain important in general, though, and it would be really nice to give at least the option to use them.

[wilkerlucio]

I agree with @cigitia, I often find myself wanting these singletons in my app, I would really like to see those in Datascript.