Skip to content

POST requests to wfs server failing #348

Closed
Closed
POST requests to wfs server failing#348
Labels
bugSomething isn't workingwfs

Description

@ateucher
ateucher
opened on Dec 18, 2024

FWIW I am experiencing the same errors using the previous and latest package versions (0.4.1 and 0.5.0).

Looks to only be the queries to the WFS server.

bcdata::bcdc_get_record("78ec5279-4534-49a1-97e8-9d315936f08b")
B.C. Data Catalogue Record: School Districts of BC
Name: school-districts-of-bc (ID:
 78ec5279-4534-49a1-97e8-9d315936f08b)
Permalink:
 https://catalogue.data.gov.bc.ca/dataset/78ec5279-4534-49a1-97e8-9d315936f08b
Licence: Open Government Licence - British Columbia
Description: The School Districts dataset contains the
 spatial representation (polygon) of the current extent of the
 administrative areas defined under section 176(1) of the School Act for
 the purposes of preservation and promotion of the fundamental principle
 of local autonomy and control of public education at the public and
 governmental levels through locally elected school boards.
Available Resources (1):
 1. WMS getCapabilities request (wms)
Access the full 'Resources' data frame using:
 bcdc_tidy_resources('78ec5279-4534-49a1-97e8-9d315936f08b')
Query and filter this data using:
 bcdc_query_geodata('78ec5279-4534-49a1-97e8-9d315936f08b')

bcdc_query_geodata('78ec5279-4534-49a1-97e8-9d315936f08b')
Error: There was an issue sending this WFS request
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
Request:
  URL: https://openmaps.gov.bc.ca/geo/pub/wfs
  POST fields:
    SERVICE=WFS&VERSION=2.0.0&REQUEST=DescribeFeatureType&typeNames=WHSE_TANTALIS.TA_SCHOOL_DISTRICTS_SVW
  Content-Type: application/x-www-form-urlencoded
  Accept-Encoding: gzip, deflate
  Accept: application/json, text/xml, application/xml, */*
  User-Agent: https://github.com/bcgov/bcdata
Response:
  status: HTTP/1.1 500 Internal Server Error
  content-type: application/json; charset=utf-8
  transfer-encoding: chunked
  vary: Origin
  vary: Access-Control-Request-Method
  vary: Access-Control-Request-Headers
  date: Wed, 18 Dec 2024 20:04:44 GMT
  server: uvicorn
  x-kong-upstream-latency: 3
  x-kong-proxy-late

Originally posted by @achubaty in #347

Activity

  1. ateucher
    added
    bugSomething isn't working
    wfs
    on Dec 18, 2024

  2. ateucher commented on Dec 18, 2024

    @ateucher
    ateucher
    on Dec 18, 2024
    CollaboratorAuthor

    This is a WFS issue: replicated using curl:

    curl -v -d "SERVICE=WFS&VERSION=2.0.0&REQUEST=DescribeFeatureType&typeNames=WHSE_LAND_AND_NATURAL_RESOURCE.PROT_HISTORICAL_FIRE_POLYS_SP" https://openmaps.gov.bc.ca/geo/pub/wfs

* Host openmaps.gov.bc.ca:443 was resolved.
* IPv6: (none)
* IPv4: 142.34.229.4
*   Trying 142.34.229.4:443...
* Connected to openmaps.gov.bc.ca (142.34.229.4) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=CA; ST=British Columbia; L=Victoria; O=Government of the Province of British Columbia; CN=*.data.gov.bc.ca
*  start date: Oct  1 15:16:12 2024 GMT
*  expire date: Oct 16 15:16:11 2025 GMT
*  subjectAltName: host "openmaps.gov.bc.ca" matched cert's "openmaps.gov.bc.ca"
*  issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2012 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1K
*  SSL certificate verify ok.
* using HTTP/1.x
> POST /geo/pub/wfs HTTP/1.1
> Host: openmaps.gov.bc.ca
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Length: 124
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 124 bytes
< HTTP/1.1 500 Internal Server Error
< content-type: text/plain; charset=utf-8
< content-length: 21
< vary: Origin
< vary: Access-Control-Request-Method
< vary: Access-Control-Request-Headers
< date: Wed, 18 Dec 2024 20:12:01 GMT
< server: uvicorn
< x-kong-upstream-latency: 2
< x-kong-proxy-latency: 1
<
* Connection #0 to host openmaps.gov.bc.ca left intact
Internal Server Error%

  3. smnorris commented on Dec 18, 2024

    @smnorris
    smnorris
    on Dec 18, 2024

    Can confirm no issues with GET requests made by Python bcdata.

    $ bcdata dump -v WHSE_TANTALIS.TA_SCHOOL_DISTRICTS_SVW > t.geojson
2024-12-18 12:09:30,389:INFO:bcdata.wfs: Total features requested: 59
2024-12-18 12:09:40,982:INFO:bcdata.wfs: https://openmaps.gov.bc.ca/geo/pub/wfs?service=WFS&version=2.0.0&request=GetFeature&typeName=WHSE_TANTALIS.TA_SCHOOL_DISTRICTS_SVW&outputFormat=json&SRSNAME=EPSG%3A4326&count=59

  4. ateucher commented on Dec 18, 2024

    @ateucher
    ateucher
    on Dec 18, 2024
    CollaboratorAuthor

    Thanks @smnorris - do you use POST requests in python bcdata anywhere?

  5. ateucher commented on Dec 18, 2024

    @ateucher
    ateucher
    on Dec 18, 2024
    CollaboratorAuthor

    Confirms working with GET via curl:

    curl -v "https://openmaps.gov.bc.ca/geo/pub/wfs?SERVICE=WFS&VERSION=2.0.0&REQUEST=DescribeFeatureType&typeNames=WHSE_LAND_AND_NATURAL_RESOURCE.PROT_HISTORICAL_FIRE_POLYS_SP"

* Host openmaps.gov.bc.ca:443 was resolved.
* IPv6: (none)
* IPv4: 142.34.229.4
*   Trying 142.34.229.4:443...
* Connected to openmaps.gov.bc.ca (142.34.229.4) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=CA; ST=British Columbia; L=Victoria; O=Government of the Province of British Columbia; CN=*.data.gov.bc.ca
*  start date: Oct  1 15:16:12 2024 GMT
*  expire date: Oct 16 15:16:11 2025 GMT
*  subjectAltName: host "openmaps.gov.bc.ca" matched cert's "openmaps.gov.bc.ca"
*  issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2012 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1K
*  SSL certificate verify ok.
* using HTTP/1.x
> GET /geo/pub/wfs?SERVICE=WFS&VERSION=2.0.0&REQUEST=DescribeFeatureType&typeNames=WHSE_LAND_AND_NATURAL_RESOURCE.PROT_HISTORICAL_FIRE_POLYS_SP HTTP/1.1
> Host: openmaps.gov.bc.ca
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< content-type: application/gml+xml; version=3.2
< transfer-encoding: chunked
< vary: Origin
< vary: Access-Control-Request-Method
< vary: Access-Control-Request-Headers
< content-disposition: inline; filename=pub-WHSE_LAND_AND_NATURAL_RESOURCE.PROT_HISTORICAL_FIRE_POLYS_SP.xsd
< date: Wed, 18 Dec 2024 20:14:39 GMT
< x-kong-upstream-latency: 11
< x-kong-proxy-latency: 0
<
<?xml version="1.0" encoding="UTF-8"?><xsd:schema xmlns:gml="http://www.opengis.net/gml/3.2" xmlns:pub="http://delivery.openmaps.gov.bc.ca/geo/" xmlns:wfs="http://www.opengis.net/wfs/2.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" targetNamespace="http://delivery.openmaps.gov.bc.ca/geo/">
  <xsd:import namespace="http://www.opengis.net/gml/3.2" schemaLocation="https://openmaps.gov.bc.ca/geo/schemas/gml/3.2.1/gml.xsd"/>
  <xsd:complexType name="WHSE_LAND_AND_NATURAL_RESOURCE.PROT_HISTORICAL_FIRE_POLYS_SPType">
    <xsd:complexContent>
      <xsd:extension base="gml:AbstractFeatureType">
        <xsd:sequence>
          <xsd:element maxOccurs="1" minOccurs="1" name="FIRE_NUMBER" nillable="false" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="VERSION_NUMBER" nillable="true" type="xsd:decimal"/>
          <xsd:element maxOccurs="1" minOccurs="1" name="FIRE_YEAR" nillable="false" type="xsd:decimal"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="FIRE_CAUSE" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="FIRE_LABEL" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="FIRE_SIZE_HECTARES" nillable="true" type="xsd:decimal"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="SOURCE" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="GPS_TRACK_DATE" nillable="true" type="xsd:date"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="LOAD_DATE" nillable="true" type="xsd:date"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="FIRE_DATE" nillable="true" type="xsd:date"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="CREATION_METHOD" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="FEATURE_CODE" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="SHAPE" nillable="true" type="gml:GeometryPropertyType"/>
          <xsd:element maxOccurs="1" minOccurs="1" name="OBJECTID" nillable="false" type="xsd:decimal"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="SE_ANNO_CAD_DATA" nillable="true" type="xsd:hexBinary"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="FEATURE_AREA_SQM" nillable="true" type="xsd:decimal"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="FEATURE_LENGTH_M" nillable="true" type="xsd:decimal"/>
        </xsd:sequence>
      </xsd:extension>
    </xsd:complexContent>
  </xsd:complexType>
  <xsd:element name="WHSE_LAND_AND_NATURAL_RESOURCE.PROT_HISTORICAL_FIRE_POLYS_SP" substitutionGroup="gml:AbstractFeature" type="pub:WHSE_LAND_AND_NATURAL_RESOURCE.PROT_HISTORICAL_FIRE_POLYS_SPType"/>
</xsd:schema>
* Connection #0 to host openmaps.gov.bc.ca left intact

  6. smnorris commented on Dec 18, 2024

    @smnorris
    smnorris
    on Dec 18, 2024

    I don't think so - all tests pass.

  7. ateucher commented on Dec 18, 2024

    @ateucher
    ateucher
    on Dec 18, 2024
    CollaboratorAuthor

    I've opened a ticket with DSS 🤞

  8. ateucher commented on Dec 19, 2024

    @ateucher
    ateucher
    on Dec 19, 2024
    CollaboratorAuthor

    DSS has fixed the service, POST requests now working again, verified in bcdata and on the command line

  9. ateucher
    closed this as completedon Dec 19, 2024

  10. achubaty commented on Dec 19, 2024

    @achubaty
    achubaty
    on Dec 19, 2024

    thank you for the quick fix!

  11. achubaty commented on Dec 19, 2024

    @achubaty
    achubaty
    on Dec 19, 2024

    @ateucher the issue persists for some requests but not others:

    > bcdata::bcdc_get_data("0bc73892-e41f-41d0-8d8e-828c16139337") ## works

> bcdata::bcdc_get_data("61044e1a-cd80-4ed6-9f95-907262b9910f") ## fails
Error: There was an issue sending this WFS request
2 paginated requests issued

> bcdata::bcdc_get_data("11277e35-d8be-47e4-bb1f-c38e393179c6") ## fails with more info
Error: There was an issue sending this WFS request
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
Request:
  URL: https://openmaps.gov.bc.ca/geo/pub/wfs
  POST fields:
    SERVICE=WFS&VERSION=2.0.0&REQUEST=DescribeFeatureType&typeNames=WHSE_LAND_USE_PLANNING.RMP_LANDSCAPE_UNIT_SVW
  Content-Type: application/x-www-form-urlencoded
  Accept-Encoding: gzip, deflate
  Accept: application/json, text/xml, application/xml, */*
  User-Agent: https://github.com/bcgov/bcdata
Response:
  status: HTTP/1.1 500 Internal Server Error
  content-type: application/json
  transfer-encoding: chunked
  vary: Origin
  vary: Access-Control-Request-Method
  vary: Access-Control-Request-Headers
  x-kong-upstream-latency: 30002
  x-kong-proxy-latency: 0
  12. stephhazlitt
    reopened this on Dec 19, 2024

  13. stephhazlitt commented on Dec 19, 2024

    @stephhazlitt
    stephhazlitt
    on Dec 19, 2024
    Member

    @achubaty I cannot reproduce the error

    bcdata::bcdc_query_geodata("61044e1a-cd80-4ed6-9f95-907262b9910f")
#> Querying 'natural-disturbance-type-map' record
#> • Using collect() on this object will return 15666 features and 8 fields
#> • Accessing this record requires pagination and will make 2 separate
#> • requests to the WFS. See ?bcdc_options
#> • At most six rows of the record are printed here
#> ────────────────────────────────────────────────────────────────────────────────
#> Simple feature collection with 6 features and 8 fields
#> Geometry type: POLYGON
#> Dimension:     XY
#> Bounding box:  xmin: 902336.9 ymin: 995587.5 xmax: 953437.5 ymax: 1255038
#> Projected CRS: NAD83 / BC Albers
#> # A tibble: 6 × 9
#>   id            FEATURE_CLASS_SKEY NATURAL_DISTURBANCE_…¹ NATURAL_DISTURBANCE_…²
#>   <chr>                      <int> <chr>                  <chr>                 
#> 1 WHSE_FOREST_…                435 NDT2                   Ecosystems with infre…
#> 2 WHSE_FOREST_…                435 NDT2                   Ecosystems with infre…
#> 3 WHSE_FOREST_…                435 NDT2                   Ecosystems with infre…
#> 4 WHSE_FOREST_…                435 NDT2                   Ecosystems with infre…
#> 5 WHSE_FOREST_…                435 NDT2                   Ecosystems with infre…
#> 6 WHSE_FOREST_…                435 NDT2                   Ecosystems with infre…
#> # ℹ abbreviated names: ¹​NATURAL_DISTURBANCE_TYPE_CODE,
#> #   ²​NATURAL_DISTURBANCE_TYPE_DESC
#> # ℹ 5 more variables: FEATURE_AREA_SQM <dbl>, FEATURE_LENGTH_M <dbl>,
#> #   OBJECTID <int>, SE_ANNO_CAD_DATA <chr>, geometry <POLYGON [m]>
    > session_info()
─ Session info ─────────────────────────────────────────
 setting  value
 version  R version 4.4.2 (2024-10-31)
 os       macOS Sonoma 14.4.1
 system   aarch64, darwin20
 ui       RStudio
 language (EN)
 collate  en_US.UTF-8
 ctype    en_US.UTF-8
 tz       America/Vancouver
 date     2024-12-19
 rstudio  2024.12.0+467 Kousa Dogwood (desktop)
 pandoc   3.2 @ /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)

─ Packages ─────────────────────────────────────────────
 package     * version  date (UTC) lib source
 bcdata      * 0.5.0    2024-12-12 [1] CRAN (R 4.4.1)

  14. achubaty commented on Dec 19, 2024

    @achubaty
    achubaty
    on Dec 19, 2024

    Sorry, the bcdc_query_geodata() call works but not the corresponding call to bcdc_get_data(). Apologies for the copy-paste error in my post above.

  15. stephhazlitt commented on Dec 19, 2024

    @stephhazlitt
    stephhazlitt
    on Dec 19, 2024
    Member

    Sorry, second example also not working for me:

    > bcdata::bcdc_get_data("11277e35-d8be-47e4-bb1f-c38e393179c6")
Error: There was an issue sending this WFS request
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
Request:
  URL: https://openmaps.gov.bc.ca/geo/pub/wfs
  POST fields:
    SERVICE=WFS&VERSION=2.0.0&REQUEST=GetFeature&outputFormat=application%2Fjson&typeNames=WHSE_LAND_USE_PLANNING.RMP_LANDSCAPE_UNIT_SVW&SRSNAME=EPSG%3A3005
  Content-Type: application/x-www-form-urlencoded
  Accept-Encoding: gzip, deflate
  Accept: application/json, text/xml, application/xml, */*
  User-Agent: https://github.com/bcgov/bcdata
Response:
  status: HTTP/1.1 504 Gateway Time-out
  date: Thu, 19 Dec 2024 18:16:38 GMT
  content-type: application/json; charset=utf-8
  transfer-encoding: chunked
  x-kong-upstream-latency: 60002
  x-kong-proxy-latency: 1

    I have re-opened the ticket with the bcgov Data Systems & Services team as this issue is with the WFS.

  16. bevingtona commented on Dec 19, 2024

    @bevingtona
    bevingtona
    on Dec 19, 2024

    All three examples work using bcdata::bcdc_query_geodata()

    bcdata::bcdc_query_geodata("0bc73892-e41f-41d0-8d8e-828c16139337") 
#> You are accessing a simplified view of the data - see the catalogue record for details.
#> Querying 'natural-resource-nr-district' record
#> • Using collect() on this object will return 23 features and 12 fields
#> • At most six rows of the record are printed here
#> ────────────────────────────────────────────────────────────────────────────────
#> Simple feature collection with 6 features and 12 fields
#> Geometry type: POLYGON
#> Dimension:     XY
#> Bounding box:  xmin: 831051 ymin: 455592.3 xmax: 1556586 ymax: 1522387
#> Projected CRS: NAD83 / BC Albers
#> # A tibble: 6 × 13
#>   id                        DISTRICT_NAME ORG_UNIT ORG_UNIT_NAME REGION_ORG_UNIT
#>   <chr>                     <chr>         <chr>    <chr>         <chr>          
#> 1 WHSE_ADMIN_BOUNDARIES.AD… Peace Natura… DPC      Peace Natura… RNO            
#> 2 WHSE_ADMIN_BOUNDARIES.AD… Cascades Nat… DCS      Cascades Nat… RTO            
#> 3 WHSE_ADMIN_BOUNDARIES.AD… Cariboo-Chil… DCC      Cariboo-Chil… RCB            
#> 4 WHSE_ADMIN_BOUNDARIES.AD… Sunshine Coa… DSC      Sunshine Coa… RSC            
#> 5 WHSE_ADMIN_BOUNDARIES.AD… Okanagan Shu… DOS      Okanagan Shu… RTO            
#> 6 WHSE_ADMIN_BOUNDARIES.AD… Stuart Necha… DVA      Stuart Necha… ROM            
#> # ℹ 8 more variables: REGION_ORG_UNIT_NAME <chr>, FEATURE_CODE <chr>,
#> #   FEATURE_NAME <chr>, OBJECTID <int>, SE_ANNO_CAD_DATA <chr>,
#> #   FEATURE_AREA_SQM <dbl>, FEATURE_LENGTH_M <dbl>, geometry <POLYGON [m]>
bcdata::bcdc_query_geodata("61044e1a-cd80-4ed6-9f95-907262b9910f")
#> Querying 'natural-disturbance-type-map' record
#> • Using collect() on this object will return 15666 features and 8 fields
#> • Accessing this record requires pagination and will make 2 separate
#> • requests to the WFS. See ?bcdc_options
#> • At most six rows of the record are printed here
#> ────────────────────────────────────────────────────────────────────────────────
#> Simple feature collection with 6 features and 8 fields
#> Geometry type: POLYGON
#> Dimension:     XY
#> Bounding box:  xmin: 902336.9 ymin: 995587.5 xmax: 953437.5 ymax: 1255038
#> Projected CRS: NAD83 / BC Albers
#> # A tibble: 6 × 9
#>   id            FEATURE_CLASS_SKEY NATURAL_DISTURBANCE_…¹ NATURAL_DISTURBANCE_…²
#>   <chr>                      <int> <chr>                  <chr>                 
#> 1 WHSE_FOREST_…                435 NDT2                   Ecosystems with infre…
#> 2 WHSE_FOREST_…                435 NDT2                   Ecosystems with infre…
#> 3 WHSE_FOREST_…                435 NDT2                   Ecosystems with infre…
#> 4 WHSE_FOREST_…                435 NDT2                   Ecosystems with infre…
#> 5 WHSE_FOREST_…                435 NDT2                   Ecosystems with infre…
#> 6 WHSE_FOREST_…                435 NDT2                   Ecosystems with infre…
#> # ℹ abbreviated names: ¹​NATURAL_DISTURBANCE_TYPE_CODE,
#> #   ²​NATURAL_DISTURBANCE_TYPE_DESC
#> # ℹ 5 more variables: FEATURE_AREA_SQM <dbl>, FEATURE_LENGTH_M <dbl>,
#> #   OBJECTID <int>, SE_ANNO_CAD_DATA <chr>, geometry <POLYGON [m]>
bcdata::bcdc_query_geodata("11277e35-d8be-47e4-bb1f-c38e393179c6") 
#> Querying 'landscape-units-of-british-columbia-current' record
#> • Using collect() on this object will return 1922 features and 20 fields
#> • At most six rows of the record are printed here
#> ────────────────────────────────────────────────────────────────────────────────
#> Simple feature collection with 6 features and 20 fields
#> Geometry type: POLYGON
#> Dimension:     XY
#> Bounding box:  xmin: 273374.5 ymin: 1562600 xmax: 715407.1 ymax: 1735705
#> Projected CRS: NAD83 / BC Albers
#> # A tibble: 6 × 21
#>   id               LANDSCAPE_UNIT_ID LANDSCAPE_UNIT_PROVID LANDSCAPE_UNIT_NUMBER
#>   <chr>                        <int>                 <int> <chr>                
#> 1 WHSE_LAND_USE_P…                 1                  1256 <NA>                 
#> 2 WHSE_LAND_USE_P…                 2                   712 <NA>                 
#> 3 WHSE_LAND_USE_P…                 3                  1324 <NA>                 
#> 4 WHSE_LAND_USE_P…                 4                    45 <NA>                 
#> 5 WHSE_LAND_USE_P…                 5                   468 <NA>                 
#> 6 WHSE_LAND_USE_P…                 6                  1268 <NA>                 
#> # ℹ 17 more variables: LANDSCAPE_UNIT_NAME <chr>,
#> #   BIODIVERSITY_EMPHASIS_OPTION <chr>, BEO_SUB_TYPE_APPLICABLE <chr>,
#> #   ORIGINAL_DECISION_DATE <chr>, LAST_AMENDMENT_DATE <chr>,
#> #   GIS_CHANGE_DATE <date>, GIS_CHANGE_PERSON <chr>, CHANGE_REASON <chr>,
#> #   INITIATOR_OF_CHANGE <chr>, ENABLING_DOCUMENT_TITLE <chr>,
#> #   ENABLING_DOCUMENT_URL <chr>, FEATURE_CODE <chr>, FEATURE_AREA_SQM <dbl>,
#> #   FEATURE_LENGTH_M <dbl>, OBJECTID <int>, SE_ANNO_CAD_DATA <chr>, …

    Created on 2024-12-19 with reprex v2.0.2

  17. stephhazlitt commented on Dec 19, 2024

    @stephhazlitt
    stephhazlitt
    on Dec 19, 2024
    Member

    Sorry, the bcdc_query_geodata() call works but not the corresponding call to bcdc_get_data(). Apologies for the copy-paste error in my post above.

    The bcdc_get_data() example is working for me, have you updated your bcdata package to the new release @achubaty ?

    bcdata::bcdc_get_data("0bc73892-e41f-41d0-8d8e-828c16139337")
#> You are accessing a simplified view of the data - see the catalogue record for details.
#> Simple feature collection with 23 features and 12 fields
#> Geometry type: MULTIPOLYGON
#> Dimension:     XY
#> Bounding box:  xmin: 273366.8 ymin: 359771.8 xmax: 1870587 ymax: 1735721
#> Projected CRS: NAD83 / BC Albers
#> # A tibble: 23 × 13
#>    id                       DISTRICT_NAME ORG_UNIT ORG_UNIT_NAME REGION_ORG_UNIT
#>  * <chr>                    <chr>         <chr>    <chr>         <chr>          
#>  1 WHSE_ADMIN_BOUNDARIES.A… Peace Natura… DPC      Peace Natura… RNO            
#>  2 WHSE_ADMIN_BOUNDARIES.A… Cascades Nat… DCS      Cascades Nat… RTO            
#>  3 WHSE_ADMIN_BOUNDARIES.A… Cariboo-Chil… DCC      Cariboo-Chil… RCB            
#>  4 WHSE_ADMIN_BOUNDARIES.A… Sunshine Coa… DSC      Sunshine Coa… RSC            
#>  5 WHSE_ADMIN_BOUNDARIES.A… Okanagan Shu… DOS      Okanagan Shu… RTO            
#>  6 WHSE_ADMIN_BOUNDARIES.A… Stuart Necha… DVA      Stuart Necha… ROM            
#>  7 WHSE_ADMIN_BOUNDARIES.A… Quesnel Natu… DQU      Quesnel Natu… RCB            
#>  8 WHSE_ADMIN_BOUNDARIES.A… Chilliwack N… DCK      Chilliwack N… RSC            
#>  9 WHSE_ADMIN_BOUNDARIES.A… Fort Nelson … DFN      Fort Nelson … RNO            
#> 10 WHSE_ADMIN_BOUNDARIES.A… Prince Georg… DPG      Prince Georg… ROM            
#> # ℹ 13 more rows
#> # ℹ 8 more variables: REGION_ORG_UNIT_NAME <chr>, FEATURE_CODE <chr>,
#> #   FEATURE_NAME <chr>, OBJECTID <int>, SE_ANNO_CAD_DATA <chr>,
#> #   FEATURE_AREA_SQM <dbl>, FEATURE_LENGTH_M <dbl>, geometry <MULTIPOLYGON [m]>

  18. achubaty commented on Dec 19, 2024

    @achubaty
    achubaty
    on Dec 19, 2024

    Yes, I am using v0.5.0.

    All three examples work with bcdc_query_geodata() alone, but the latter 2 fail with additional collect() call. All three examples fail using bcdc_get_data() calls.

    > bcdata::bcdc_get_data("0bc73892-e41f-41d0-8d8e-828c16139337") ## works
> bcdata::bcdc_get_data("61044e1a-cd80-4ed6-9f95-907262b9910f") ## fails
> bcdata::bcdc_get_data("11277e35-d8be-47e4-bb1f-c38e393179c6") ## fails

  19. ateucher commented on Dec 19, 2024

    @ateucher
    ateucher
    on Dec 19, 2024
    CollaboratorAuthor

    It looks like it is still an issue with POST requests, but only those requiring pagination using sortby, count and startIndex parameters:

    Works with GET:

    $ curl https://openmaps.gov.bc.ca/geo/pub/wfs?SERVICE=WFS&VERSION=2.0.0&REQUEST=GetFeature&outputFormat=application%2Fjson&typeNames=WHSE_FOREST_VEGETATION.BEC_NATURAL_DISTURBANCE_SV&SRSNAME=EPSG%3A3005&sortby=OBJECTID&startIndex=0&count=10000 -o bec1.json

    Not with POST:

    $ curl -v -d "SERVICE=WFS&VERSION=2.0.0&REQUEST=GetFeature&outputFormat=application%2Fjson&typeNames=WHSE_FOREST_VEGETATION.BEC_NATURAL_DISTURBANCE_SV&SRSNAME=EPSG%3A3005&sortby=OBJECTID&startIndex=0&count=10000" https://openmaps.gov.bc.ca/geo/pub/wfs
* Host openmaps.gov.bc.ca:443 was resolved.
* IPv6: (none)
* IPv4: 142.34.229.4
*   Trying 142.34.229.4:443...
* Connected to openmaps.gov.bc.ca (142.34.229.4) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=CA; ST=British Columbia; L=Victoria; O=Government of the Province of British Columbia; CN=*.data.gov.bc.ca
*  start date: Oct  1 15:16:12 2024 GMT
*  expire date: Oct 16 15:16:11 2025 GMT
*  subjectAltName: host "openmaps.gov.bc.ca" matched cert's "openmaps.gov.bc.ca"
*  issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2012 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1K
*  SSL certificate verify ok.
* using HTTP/1.x
> POST /geo/pub/wfs HTTP/1.1
> Host: openmaps.gov.bc.ca
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Length: 197
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 197 bytes
< HTTP/1.1 500 Internal Server Error
< content-type: application/json
< transfer-encoding: chunked
< vary: Origin
< vary: Access-Control-Request-Method
< vary: Access-Control-Request-Headers
< x-kong-upstream-latency: 24408
< x-kong-proxy-latency: 0
<
* Connection #0 to host openmaps.gov.bc.ca left intact
{"message":"Upstream service error - An unexpected error occurred","upstream":{"requestId":"6043c512-1010262","timestamp":"2024-12-19T20:00:12.800+00:00","path":"/geo/pub/wfs","status":500,"error":"Internal Server Error"}}%

  20. ateucher commented on Dec 19, 2024

    @ateucher
    ateucher
    on Dec 19, 2024
    CollaboratorAuthor

    It looks like it is timing out with bigger requests.

    This worked: count=100, startIndex=20

    curl -d "SERVICE=WFS&VERSION=2.0.0&REQUEST=GetFeature&typeNames=WHSE_FOREST_VEGETATION.BEC_NATURAL_DISTURBANCE_SV&OutputFormat=application/json&sortby=OBJECTID&StartIndex=20&count=100" https://openmaps.gov.bc.ca/geo/pub/wfs > dev/null

    This timed out: count=5666, startIndex=10000

    $ curl -v -d "SERVICE=WFS&VERSION=2.0.0&REQUEST=GetFeature&typeNames=WHSE_FOREST_VEGETATION.BEC_NATURAL_DISTURBANCE_SV&OutputFormat=application/json&sortby=OBJECTID&StartIndex=10000&count=5666" https://openmaps.gov.bc.ca/geo/pub/wfs > dev/null

* Host openmaps.gov.bc.ca:443 was resolved.
* IPv6: (none)
* IPv4: 142.34.229.4
*   Trying 142.34.229.4:443...
* Connected to openmaps.gov.bc.ca (142.34.229.4) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=CA; ST=British Columbia; L=Victoria; O=Government of the Province of British Columbia; CN=*.data.gov.bc.ca
*  start date: Oct  1 15:16:12 2024 GMT
*  expire date: Oct 16 15:16:11 2025 GMT
*  subjectAltName: host "openmaps.gov.bc.ca" matched cert's "openmaps.gov.bc.ca"
*  issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2012 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1K
*  SSL certificate verify ok.
* using HTTP/1.x
> POST /geo/pub/wfs HTTP/1.1
> Host: openmaps.gov.bc.ca
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Length: 178
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 178 bytes
< HTTP/1.1 504 Gateway Time-out
< date: Thu, 19 Dec 2024 20:18:44 GMT
< content-type: application/json; charset=utf-8
< transfer-encoding: chunked
< x-kong-upstream-latency: 60002
< x-kong-proxy-latency: 1
<
* Connection #0 to host openmaps.gov.bc.ca left intact
{"message":"Upstream service error - The upstream server is timing out","upstream":{"message":"The upstream server is timing out"}}%

  21. ateucher commented on Dec 20, 2024

    @ateucher
    ateucher
    on Dec 20, 2024
    CollaboratorAuthor

    This appears to be fixed on the server side now, @achubaty, @bevingtona if you don't mind doing a quick test that would be great!

  22. achubaty commented on Dec 20, 2024

    @achubaty
    achubaty
    on Dec 20, 2024

    Looks like it's working again (albeit slowly). Thank you!

  23. ateucher commented on Dec 20, 2024

    @ateucher
    ateucher
    on Dec 20, 2024
    CollaboratorAuthor

    Yeah, I found it a bit slow as well... thanks for testing!

  24. ateucher
    closed this as completedon Dec 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingwfs

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions