Repository navigation
Public dataset for testing #9
Description
@tucotuco, @laurarussell: CSV file with Darwin Core data for public testing that's representative of real world data.
{{ message }}
@tucotuco, @laurarussell: CSV file with Darwin Core data for public testing that's representative of real world data.
Activity
tucotuco commentedon Aug 1, 2011
@eightysteele Laura will give you some "unclean" data for this. Do you want to start loading up real data in real collections as well? Hold off? What are the immediate goals on the data store side?
eightysteele commentedon Aug 1, 2011
For now let's just upload our data sets (public and internal) to the VertNet collection on Google Docs: http://goo.gl/xSMka
tucotuco commentedon Aug 1, 2011
Clarification. @laurarussell Please put the uncleaned data set as a csv with a Darwin Core header into the VertNet/Dev/Data collection on Google Docs, located at the link in the previous comment. Call it Test_[InstitutionCode]_[CollectionCode].csv, substituting the actual InstitutionCode and CollectionCode into the filename and omitting the square brackets. We will not assemble any other data sets at this time.
laurarussell commentedon Aug 4, 2011
Added test_NYSM_ZM_data.csv and test_NYSM_ZM_README.txt to the data folder on Google Docs. I'll continue to add more.
eightysteele commentedon Aug 4, 2011
+1
eightysteele commentedon Aug 4, 2011
@laurarussell: Actually, for test data sets right now, can we add a column for
occurrenceid?laurarussell commentedon Aug 4, 2011
@tucotuco is it okay to just rename the CatalogNumberNumeric field to occurrenceid? Otherwise I can try and add it via Excel, but I've been very careful not to mess with the structure of this file so as not to lose the line feeds.
tucotuco commentedon Aug 5, 2011
In this case yes, we can. If they are not unique, that's actually a good part of the test. @eightysteele This is actually a mis-use of the Darwin Core term occurrenceID, which is supposed to be a persistent resolvable GUID, and the reason GBIF called it the sourceID in the IPT. We may want to add similar logic. That is, if there is an occurrenceID in the source, then use it not only as the dwc:occurrenceID, but also as a unique identifier within the collection, or sourceID.
eightysteele commentedon Aug 5, 2011
@tucotuco: See issue #1 for background on our thinking about
sourceid. Basically any column in the CSV can be specified by the publisher to be thesourceid. Of course, ifsourceidis a column they don't have to specify it. Usingoccurrenceidis just a short term hack until #1 is complete.laurarussell commentedon Aug 5, 2011
@eightysteele @tucotuco - uploaded a new version with CatalogNumberNumerica now as occurrenceID. (Didn't check yet to see if there are any dups in there though.)
laurarussell commentedon Aug 5, 2011
whoops, didn't mean to close issue. I thought comment and close meant it would close my tab. Sorry :-(
eightysteele commentedon Aug 5, 2011
:)
eightysteele commentedon Aug 5, 2011
Ok, I'm bulkloading the test dataset to App Engine now. Closing issue. Thanks @laurarussell!
eightysteele commentedon Aug 18, 2011
Reopening issue. @tucotuco and @laurarussell, can we create a similar data set with all Darwin Core fields populated? This will help us in testing the API. Thoughts?
laurarussell commentedon Aug 31, 2011
Sorry @eightysteele and @tucotuco, just now seeing this. I was out Aug. 17-19 and missed this last week. Are you still needing a full field data set? KU Birds might be close to having all fields although not all are populated. Not sure we actually have one with all fields populated. Also, I was helping Specify staff with a CU Paleo file so now have a text file of that if we need a paleo one down the road. @tucotuco Was thinking Cambridge might be a good data set with mixed class records in one data set.
tucotuco commentedon Aug 31, 2011
@laurarussell The key here for testing purposes is to have a dataset as fully populated in DwC fields as possible. It could be the KU set, if that is the closest we have, but adding data for every possible DwC field that might not be in the database (type, basisOfRecord, rights, etc. etc.). I wouldn't worry about coming up with things like identifiers (taxonID, identificationID, etc.) that don't exist.
eightysteele commentedon Aug 31, 2011
Yeah, I think we should just start adding test data to our Google Docs collection so that we have a large corpus of records to experiment with. No rush though, but the sets you describe above sound pretty rad, especially the CU Paleo stuff!