How to successfully load the GoogleNews-vectors-negative300 model?

[marcoippolito]

Hi Philipp,
I downloaded from https://code.google.com/p/word2vec/ the file GoogleNews-vectors-negative300.bin.gz

w2v = require('word2vec');
{ word2vec: [Function: word2vec],
word2phrase: [Function: word2phrase],
loadModel: [Function: loadModel],
WordVector: [Function: WordVector] }

w2v.loadModel("/home/marco/crawlscrape/bashUtilitiesDir/GoogleNews-vectors-negative300.bin", function(err, model) {
... console.log(model);
... });
undefined
TypeError: Cannot read property 'length' of undefined
at /home/marco/node_modules/word2vec/lib/model.js:408:30
at FSReqWrap.wrapper as oncomplete

w2v.loadModel("/home/marco/crawlscrape/bashUtilitiesDir/GoogleNews-vectors-negative300.bin", function(err, model) {
... console.log(model);
... });
undefined
TypeError: undefined is not a function
at readOne (/home/marco/node_modules/word2vec/lib/model.js:433:55)
at FSReqWrap.wrapper as oncomplete

What do I have to do in order to successfully load the GoogleNews-vectors-negative300 model?

Looking forward to your kind help.
Marco

[dariusk]

I tried doing this last week -- I'm pretty sure that it doesn't accept trained models in the binary (.bin) format, only in text format. While it's possible to convert the binary format to text, the resulting model is so big that it caused Node.js to run out of memory while consuming it. (This is independent of the RAM of the machine and has more to do with the limitations on address space on a 64-bit CPU.)

[marcoippolito]

Hi. If the only accepted format is text format, with which the resulting model of GoogleNews-vectors-negative300.bin is so big that it causes Node.js to run out of memory while consuming it, this module, despite being potentially very usefull in many situations, cannot now be deployed and used.
The best would be, as I do with a Python module, to directly load the compressed file, GoogleNews-vectors-negative300.bin.gz, in order to speed up the loading activity, and save some memory (a scarse resource, even for powerful machines).....What do you think Philipp?

[Planeshifter]

I don't have a good Internet connection right now (using my phone as a router), but will look into this later this afternoon.

[Planeshifter]

Okay, I made some little changes to the code. Could you please clone the Github repo, run npm install and try again? Cannot check as I do not have the corpus available right now.

[marcoippolito]

Hi Philipp,
tell me please what I'm doing wrong...

marco@pc:/node_modules$ git clone https://github.com/Planeshifter/node-word2vec.git
Cloning into 'node-word2vec'...
remote: Counting objects: 329, done.
remote: Total 329 (delta 0), reused 0 (delta 0), pack-reused 329
Ricezione degli oggetti: 100% (329/329), 283.85 KiB | 355.00 KiB/s, done.
Risoluzione dei delta: 100% (175/175), done.
Checking connectivity... fatto.
marco@pc:/node_modules$ cd node-word2vec
marco@pc:~/node_modules/node-word2vec$ ls -a
. .. data .editorconfig examples .git .gitignore .jshintignore .jshintrc lib LICENSE .npmignore package.json README.md src test .travis.yml

sudo npm install
npm WARN package.json ggc@0.0.1 No README data
npm WARN package.json ggc@0.0.1 No bin file found at ./bin/http-server
npm WARN cannot run in wd word2vec@0.9.1 make --directory=src (wd=/home/marco/node_modules/node-word2vec)

[Planeshifter]

It seems that one needs to set the flag unsafe-perm=true to run npm scripts as root, so your sudo was causing this issue. I pushed a little fix such that your code should work now. Could you try again? Thanks,
Philipp

[marcoippolito]

marco@pc:/node_modules$ rm -rf node-word2vec
marco@pc:/node_modules$ git clone https://github.com/Planeshifter/node-word2vec.git
Cloning into 'node-word2vec'...
remote: Counting objects: 333, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 333 (delta 0), reused 0 (delta 0), pack-reused 329
Ricezione degli oggetti: 100% (333/333), 285.03 KiB | 0 bytes/s, done.
Risoluzione dei delta: 100% (175/175), done.
Checking connectivity... fatto.

marco@pc:$ sudo npm install
[sudo] password for marco:
npm WARN package.json ggc@0.0.1 No README data
npm WARN package.json ggc@0.0.1 No bin file found at ./bin/http-server
npm WARN cannot run in wd word2vec@0.9.1 make --directory=src (wd=/home/marco/node_modules/node-word2vec)
marco@pc:$ sudo npm install unsafe-perm=true
npm WARN package.json ggc@0.0.1 No README data
npm WARN package.json ggc@0.0.1 No bin file found at ./bin/http-server
npm ERR! addLocal Could not install /home/marco/unsafe-perm=true
npm ERR! Linux 3.13.0-32-generic
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "install" "unsafe-perm=true"
npm ERR! node v0.12.7
npm ERR! npm v2.11.3
npm ERR! path /home/marco/unsafe-perm=true
npm ERR! code ENOENT
npm ERR! errno -2

npm ERR! enoent ENOENT, open '/home/marco/unsafe-perm=true'
npm ERR! enoent This is most likely not a problem with npm itself
npm ERR! enoent and is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! Please include the following file with any support request:
npm ERR! /home/marco/npm-debug.log

[dariusk]

I'm traveling but I'll definitely give this a shot this weekend.

[marcoippolito]

I did this:
npm install
npm WARN package.json ggc@0.0.1 No README data
npm WARN package.json ggc@0.0.1 No bin file found at ./bin/http-server

word2vec@0.9.1 postinstall /home/marco/node_modules/node-word2vec
make --directory=src

make: ingresso nella directory "/home/marco/node_modules/node-word2vec/src"
gcc word2vec.c -o word2vec -lm -pthread -O3 -march=native -Wall -funroll-loops -Wno-unused-result -fno-stack-protector
gcc word2phrase.c -o word2phrase -lm -pthread -O3 -march=native -Wall -funroll-loops -Wno-unused-result -fno-stack-protector
gcc distance.c -o distance -lm -pthread -O3 -march=native -Wall -funroll-loops -Wno-unused-result -fno-stack-protector
distance.c: In function ‘main’:
distance.c:31:8: warning: unused variable ‘ch’ [-Wunused-variable]
char ch;
^
gcc word-analogy.c -o word-analogy -lm -pthread -O3 -march=native -Wall -funroll-loops -Wno-unused-result -fno-stack-protector
word-analogy.c: In function ‘main’:
word-analogy.c:31:8: warning: unused variable ‘ch’ [-Wunused-variable]
char ch;
^
gcc compute-accuracy.c -o compute-accuracy -lm -pthread -O3 -march=native -Wall -funroll-loops -Wno-unused-result -fno-stack-protector
compute-accuracy.c: In function ‘main’:
compute-accuracy.c:28:109: warning: unused variable ‘ch’ [-Wunused-variable]
char st1[max_size], st2[max_size], st3[max_size], st4[max_size], bestw[N][max_size], file_name[max_size], ch;
^
chmod +x *.sh
make: uscita dalla directory "/home/marco/node_modules/node-word2vec/src"
marco@pc:~$ node

w2v = require('word2vec');
Error: Cannot find module 'word2vec'
at Function.Module._resolveFilename (module.js:336:15)
at Function.Module._load (module.js:278:25)
at Module.require (module.js:365:17)
at require (module.js:384:17)
at repl:1:7
at REPLServer.defaultEval (repl.js:132:27)
at bound (domain.js:254:14)
at REPLServer.runBound as eval
at REPLServer. (repl.js:279:12)
at REPLServer.emit (events.js:107:17)

npm install unsafe-perm=true
npm WARN package.json ggc@0.0.1 No README data
npm WARN package.json ggc@0.0.1 No bin file found at ./bin/http-server
npm ERR! addLocal Could not install /home/marco/unsafe-perm=true
npm ERR! Linux 3.13.0-32-generic
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "install" "unsafe-perm=true"
npm ERR! node v0.12.7
npm ERR! npm v2.11.3
npm ERR! path /home/marco/unsafe-perm=true
npm ERR! code ENOENT
npm ERR! errno -2

npm ERR! enoent ENOENT, open '/home/marco/unsafe-perm=true'
npm ERR! enoent This is most likely not a problem with npm itself
npm ERR! enoent and is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! Please include the following file with any support request:
npm ERR! /home/marco/npm-debug.log

What do I have to do Philipp?

[Planeshifter]

Hmm, solving this problem seems to be more complicated than I had hoped. If you want to have a look yourself, the code to read binary files is located in the function readBinary in model.js. This code was generously contributed by @oskarflordal and is not written by myself. One of the errors @marcoippolito ran into was caused by the fact that as of node version v0.12, typed arrays do not possess a slice method anymore.

And somehow in the GoogleNews data set all words are missing their first characters when extracted from the binary data, the likely cause for the TypeError: Cannot read property 'length' of undefined error.

After having run a bunch of tests, it seems that right now the code does not correctly read in the vector values from the binary data, either. Oskar, if you find the time, could you have a look?

I need to look into this when I have more time. I fear this won't be resolved in a short amount of time, unfortunately.

[Planeshifter]

Just published a new version of the package to npm with some changes in the readBinary function. Could you try installing it as usual via npm install word2vec and then loading the GoogleNews corpus again? My Laptop does not handle the large file size of 3.5GB, so I cannot check whether the problem is solved. Thanks!

[marcoippolito]

git clone https://github.com/Planeshifter/node-word2vec.git
Cloning into 'node-word2vec'...
remote: Counting objects: 349, done.
remote: Compressing objects: 100% (23/23), done.
remote: Total 349 (delta 9), reused 0 (delta 0), pack-reused 325
Ricezione degli oggetti: 100% (349/349), 294.95 KiB | 0 bytes/s, done.
Risoluzione dei delta: 100% (181/181), done.
Checking connectivity... fatto.

sudo npm install
[sudo] password for marco:
npm WARN package.json ggc@0.0.1 No README data
npm WARN package.json ggc@0.0.1 No bin file found at ./bin/http-server
npm WARN cannot run in wd word2vec@0.9.2 make --directory=src (wd=/home/marco/node_modules/node-word2vec)
marco@pc:~$ npm install
npm WARN package.json ggc@0.0.1 No README data
npm WARN package.json ggc@0.0.1 No bin file found at ./bin/http-server

word2vec@0.9.2 postinstall /home/marco/node_modules/node-word2vec
make --directory=src

make: ingresso nella directory "/home/marco/node_modules/node-word2vec/src"
gcc word2vec.c -o word2vec -lm -pthread -O3 -march=native -Wall -funroll-loops -Wno-unused-result -fno-stack-protector
gcc word2phrase.c -o word2phrase -lm -pthread -O3 -march=native -Wall -funroll-loops -Wno-unused-result -fno-stack-protector
gcc distance.c -o distance -lm -pthread -O3 -march=native -Wall -funroll-loops -Wno-unused-result -fno-stack-protector
distance.c: In function ‘main’:
distance.c:31:8: warning: unused variable ‘ch’ [-Wunused-variable]
char ch;
^
gcc word-analogy.c -o word-analogy -lm -pthread -O3 -march=native -Wall -funroll-loops -Wno-unused-result -fno-stack-protector
word-analogy.c: In function ‘main’:
word-analogy.c:31:8: warning: unused variable ‘ch’ [-Wunused-variable]
char ch;
^
gcc compute-accuracy.c -o compute-accuracy -lm -pthread -O3 -march=native -Wall -funroll-loops -Wno-unused-result -fno-stack-protector
compute-accuracy.c: In function ‘main’:
compute-accuracy.c:28:109: warning: unused variable ‘ch’ [-Wunused-variable]
char st1[max_size], st2[max_size], st3[max_size], st4[max_size], bestw[N][max_size], file_name[max_size], ch;
^
chmod +x *.sh
make: uscita dalla directory "/home/marco/node_modules/node-word2vec/src"

[marcoippolito]

Hi Philipp,
did you find something related?
If you to test something, I can give it a try.
Let me know.
Maroc

[marcoippolito]

Hi Philipp and hi Oskar,
the indications here: https://bassnutz.wordpress.com/2012/09/09/processing-large-files-with-nodejs/
could be of help for importing GoogleNews-vectors-negative300.bin.gz?

[Planeshifter]

Hi Marco, sorry for not following up, have been busy. Will look at your link shortly. Sorry for the delayed response. Best,
Philipp

P.S. Did you try installing the package with npm install word2vec?