Localhost data migration

[kwe]

I've Mongodb, Parse server running locally - Is there a way to import the data without going through the need to make my development DB available on the internet so that Parse can connect. This is purely to test the setup prior to a formal migration..

[docherty]

@kwe Can you export the data from Parse (you get a zip to download) and then import that into your local mongoDB? The zip won't contain any files uploaded to Parse but will include reference urls.

[gfosco]

There is a way, using a program like ngrok to expose your mongo port to the outside... But I'm not sure we support running multiple migrations... I'll check on this.

[kwe]

I did an export, unzipped and then..

Edited each file removing from the first line..

{ "results": [
and the closing '}' from the last line of the file.

(There may be a more elegant way to do this) - it was needed to avoid ending up with a single document with an embedded "results" array.

mongoimport --db dev --collection Health --file Health.json --jsonArray

seems to work. i.e. import Health resource into local mongodb

[anthonylai]

I tried doing the migration to localhost, and it worked.

I was at home, and I could set the router settings with port forwarding for 27101 to my machine. I can then provide my external ip address to the migration tool, and my data get imported.

The mongodb connection I used is: mongodb://111.111.111.111:27017/my-db-name

I tried ngrok and it doesn't work for me.

Hope this helps.

[doender]

@kwe I tried importing the data like this this, but I think it isn't stored properly. For example, querying on a given object id doesn't work.

The objects that are created through the API are represented like this:

{
    "_id" : "4I8dI7GUpE",
    "_updated_at" : ISODate("2016-02-01T10:27:49.363Z"),
    "_created_at" : ISODate("2016-02-01T10:27:49.363Z")
}

Whereas the objects that are imported from the exported Parse zip file look like this:

{
    "_id" : ObjectId("56af319ee015b5c2d0016352"),
    "objectId" : "4I8dI7GUpE",
    "updatedAt" : "2015-07-06T14:54:01.760Z",
    "createdAt" : "2015-04-08T09:44:03.741Z"
}

[kwe]

hmm. I'm new to mongodb - I guess what you're finding would break objects that reference other resources by _id. My relatively simple app works fine, will try with a more complex example. I guess I'd be tempted to globally replace 'objectId' with '_id' and try an import. Don't know if mongodb allows import to include id value like that.

[doender]

I created a Python import script that imports objects from Parse-exported JSON files (can parse big files, using the ijson module). This now does indeed add _id fields from the objectIds, and this seems to work when querying for an individual object. Would such an import script be interesting to add to this or another repository?

I have three other questions about this importer:

• How are pointers represented in a Parse compatible mongodb?
• How should dates be represented/translated in/to mongodb? Should updatedAt/createdAt be stored as _updated_at/_created_at and dates be converted from strings to ISODates?
• Are there any other things to take care of when importing the Parse-exported JSON files into a Parse compatible mongodb?

[gfosco]

Pointers are represented as an object like {"__type":"Pointer", "className":"FooBars", "objectId":"abc123"}

Indeed, _created_at and _updated_at in the database, as ISO formatted strings, as in our rest api.

There are lots of special cases, and I'd look at transform.js to see some of the crazy olympics.

[doender]

@gfosco Thanks! I inspected the code a bit further, and maybe a NodeJS importer would make most sense, based on the existing ExportAdapter.js (create and addRelation methods) and some way to read large JSON files, like https://github.com/dominictarr/JSONStream.

What do you think?
The only thing I'm not really sure about is how to set the class-based permissions (these could come from a separate input file for the importer).

[gfosco]

I would suggest against building a custom importer, and instead just use the normal migration process with an internet accessible mongo installation.

@gfosco

I'm not sure we support running multiple migrations... I'll check on this

Any word? And does running the migration process unavoidably migrate off of the hosted Parse db, or is that an optional step?

[drew-gross]

The finalize button on the migration page will stop all reads and writes to the Parse.com database and begin reading and writing to your own database exclusively. Before you have clicked finalize, you can choose to cancel the migration and start another migration to a different destination.