Repository navigation
Gotta do something with those stale downloads... #597
Description
https://console.cloud.google.com/appengine/taskqueues/compose?project=vertnet-portal
There are 18 downloads "in progress", all of them from many days ago (up to 197). We should checkout the code to see what is causing them and fix it. It is taking a lot of backend instance time (and that means $$)
Activity
tucotuco commentedon May 23, 2016
Below is a list of the queries that were hung up so that we can try to find
a pattern or try to reproduce them. I removed the tasked from the queue.
q=M%C3%A9xico+class%3AActinopterygii+country%3A%22M%C3%A9xico%22&requesttime=2016-05-06T16%3A31%3A31.001020&name=Peces.M%C3%A9xico&fileindex=0&filepattern=Peces.M%C3%A9xico-356f3b60858b41bbaef04418e2eb3b40&reccount=1000&email=gicruz%40ecosur.edu.mx
q=preparations%3A+wet+type%3Aspecimen+class%3AMammalia&requesttime=2016-02-24T22%3A14%3A07.161640&name=Thayn%C3%A1_Brito&fileindex=0&filepattern=Thayn%C3%A1_Brito-83ed7d47fb294dd68662e780e1f70706&reccount=1000&email=thay_na96%40hotmail.com
q=Brehm+type%3Aspecimen&requesttime=2015-11-05T08%3A07%3A59.809910&name=Brehm_%C3%9Cbersee&fileindex=0&filepattern=Brehm_%C3%9Cbersee-971a80f30ea2417caea1635d2d69ec35&reccount=1000&email=t.toepfer%40zfmk.de
q=specificepithet%3Ahudsonius+genus%3Azapus+locality%3AYakutat+Bay+stateprovince%3A%22Alaska%22+institutioncode%3AUSNM&requesttime=2016-02-13T23%3A06%3A59.391770&name=Zahu_al_type&fileindex=0&filepattern=Zahu_al_type-cc543119a54d42e186978ceae51266ff&reccount=1&email=
q=Selasphorus+type%3Aspecimen+mappable%3A1&requesttime=2015-12-08T19%3A50%3A30.052400&name=MyResults&fileindex=0&filepattern=MyResults-65b1e51ec7cf4f4991c0272dd735a932&reccount=1000&email=cjbattey%40uw.edu
q=Cardellina+pusilla&requesttime=2015-12-08T17%3A11%3A43.545530&name=MyResults&fileindex=0&filepattern=MyResults-55de92961bf241239cf8a2db739c3999&reccount=1000&email=ryleec999%40gmail.com
q=class%3AAmphibia+stateprovince%3AOregon&requesttime=2016-01-21T22%3A38%3A25.829150&name=ORAmphibians&fileindex=31&filepattern=ORAmphibians-f15e6813b8f14fafb802546e4489f782&reccount=31821&email=mmims%40usgs.gov
q=specificepithet%3Aprinceps+genus%3Aochotona&requesttime=2015-12-08T15%3A46%3A45.684890&name=test-bigsearch1&fileindex=0&filepattern=test-bigsearch1-e0d031980f6849d68c5e378c549370dd&reccount=1000&email=myrmecocystus%40gmail.com
q=specificepithet%3Acollaris+genus%3Aochotona+year%3A1960&requesttime=2015-12-08T15%3A46%3A45.886060&name=test-bigsearch2&fileindex=0&filepattern=test-bigsearch2-2073dee868a247f592dbb8bca913f48f&reccount=1&email=myrmecocystus%40gmail.com
q=Ecuador+amphibia&requesttime=2016-02-04T19%3A38%3A52.527740&name=Mario_H._Yanez-Mu%C3%B1oz&fileindex=0&filepattern=Mario_H._Yanez-Mu%C3%B1oz-e91ea0f24b63462bba4568d0e96d733f&reccount=1000&email=mayamu%40hotmail.com
q=Ecuador+reptilia&requesttime=2016-02-04T19%3A39%3A41.078270&name=Mario_H._Yanez-Mu%C3%B1oz&fileindex=0&filepattern=Mario_H._Yanez-Mu%C3%B1oz-096bd7c775d14ddc87a98869796e812d&reccount=1000&email=mayamu%40hotmail.com
q=family%3APhrynosomatidae+stateprovince%3A%22Nuevo+Le%C3%B3n%22+country%3A%22M%C3%A9xico%22&requesttime=2016-01-26T02%3A28%3A49.131840&name=Phrynosomatidae_Nuevo_Le%C3%B3n&fileindex=0&filepattern=Phrynosomatidae_Nuevo_Le%C3%B3n-47f10562f3fb4b49a1735293c49959b9&reccount=1000&email=manuelocampo_13%40hotmail.com
q=class%3AAmphibia+stateprovince%3AOregon&requesttime=2016-01-21T22%3A38%3A25.294150&name=ORAmphibians&fileindex=31&filepattern=ORAmphibians-3cddf114a4f94ab48b0d4f1bee18abe5&reccount=31821&email=mmims%40usgs.gov
q=class%3AAmphibia+stateprovince%3AOregon&requesttime=2016-01-21T22%3A38%3A24.878130&name=ORAmphibians&fileindex=31&filepattern=ORAmphibians-e7576d2bb2fb45b4ad0fdef5f70dfa2f&reccount=31821&email=mmims%40usgs.gov
q=class%3AAmphibia+stateprovince%3AOregon&requesttime=2016-01-21T22%3A38%3A24.878130&name=ORAmphibians&fileindex=31&filepattern=ORAmphibians-e7576d2bb2fb45b4ad0fdef5f70dfa2f&reccount=31821&email=mmims%40usgs.gov
q=Troglodytes&requesttime=2015-12-08T19%3A52%3A48.103150&name=Troglodytes&fileindex=41&filepattern=Troglodytes-788c6905ca7e47c7bf3d8943efdee04d&reccount=41572&email=barke042%40umn.edu
q=distance%28location%2Cgeopoint%28-28.071980301779845%2C-61.5234375%29%29%3C400129&requesttime=2016-03-30T14%3A09%3A55.222070&name=MyResults&fileindex=6&filepattern=MyResults-8bf52987989443cb8128aac8bad46957&reccount=6646&email=agustinomen%40hotmail.com
q=mammals+and+nayarit+%28mammals%29&requesttime=2016-04-25T18%3A27%3A31.636690&name=Mam%C3%ADferos_de_Nayarit_VERTNET&fileindex=0&filepattern=Mam%C3%ADferos_de_Nayarit_VERTNET-38a517599e3141dabed2ad5d6dee0da9&reccount=1000&email=zacatuchemx%40hotmail.com
On Fri, May 20, 2016 at 7:49 AM, Javier Otegui notifications@github.com
wrote:
tucotuco commentedon Jan 4, 2017
This remains a recurring theme. The only solution so far is a brute force one. Open the downloadwrite task queue and delete everything in it. This will affect downloads in progress. After doing so, the files in the vn-dltest bucket in Google Cloud storage have to be removed. Use the GCS_cleaner.py script to do this. https://github.com/VertNet/post-harvest-processor/tree/master/lib