Repository navigation
Add sourceid support #1
Description
The VN bulkloader requires a sourceid from which datastore keys are constructed. The sourceid is the link between records published in VN and their original version at the source.
The bulkloader needs a command line parameter that defines the column in the csv file that is the sourceid.
Activity
eightysteele commentedon Jul 28, 2011
In the absence of a sourceid command line parameter, the bulkloader should look for a sourceid column in the CSV file.
datadavev commentedon Jul 28, 2011
sourceid should be a managed value. What is the format / acceptable chars? who creates them? who assigns them? How are duplicates avoided?
One solution might be to require that the sourceid is something generated from a reputable source such as EZID?
eightysteele commentedon Jul 28, 2011
Agreed. The sourceid should be managed by the publisher.
Thinking any string 500 bytes or less (since that's the max size of an App Engine key_name).
Publishers. If the publisher doesn't provide a sourceid for a record, the bulkloader can:
Right now we're leaning towards 2. For any generated sourceid, the CSV file would be re-written to include the sourceid column.
Right, so upon detecting a duplicate, the bulkloader will simply ignore the record and report the duplicate to the user. It's then up to the publisher to fix the duplicate and re-run the bulkloader.
datadavev commentedon Jul 28, 2011
I would strongly prefer that the sourceid should be generated by a service like EZID. Then it can be reliably used to cross reference between different systems and is not tied specifically to vertnet infrastructure (one less thing for vertnet to maintain).
eightysteele commentedon Jul 28, 2011
Well, the sourceid in this context is literally the id of a record at the publisher source, like a database row id or something. In most cases it will already exist. It's not something that VertNet will generate for every record.
Aside from sourceid, we could think about assigning an EZID to each VertNet record to handle your use case. I added issue #8 for this.
datadavev commentedon Jul 28, 2011
ah, my mistake. I was interpreting it as an identifier for the collection from which the record is being pulled, not for the individual record.