Skip to content

pool: propagate SciTag to remote party in HTTP-TPC transfers - #8271

Open
chubnastia wants to merge 1 commit into
12.0from
scitag_investigation-12.0
Open

chubnastia wants to merge 1 commit into
12.0from
scitag_investigation-12.0

Conversation

@chubnastia

@chubnastia chubnastia commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Motivation:
The WebDAV door forwards the SciTag supplied with a COPY request to the
pool via RemoteHttpDataTransferProtocolInfo. The active pool uses it for
its own firefly markers and billing record, but never sends it to the
remote (passive) party. RemoteHttpDataTransferProtocol only forwards
the TransferHeader* headers, so a client sending a plain "SciTag" header
(as FTS does) results in no SciTag on the GET/PUT/HEAD/DELETE requests
issued by the pool. The passive endpoint therefore cannot classify the
flow, and its billing records (e.g. Http/Https mover records in the kafka
stream) have an empty transferTag.

Modification:
In RemoteHttpDataTransferProtocol#addHeadersToRequest, add a "SciTag"
request header carrying the transfer tag if one is set and the client
did not already supply one explicitly via "TransferHeaderSciTag". As all
requests (including retries and redirects) are built through this
method, the remote party sees the same tag for the whole transfer.

Result:
The passive endpoint of an HTTP-TPC transfer receives the same SciTag as
the active pool, so both sides record the same transferTag in billing
and can emit matching flow markers. Transfers without a tag are
unchanged.

Testing:
- system-test with billing.format.json=true, push COPY on the HTTPS
WebDAV door (port 2881) to itself:
before: active pool record transferTag=65, passive pool record
transferTag="" (door logs tagSource=none)
after: both records transferTag=65 (door logs tagSource=SciTag-header)

Motivation:
The WebDAV door forwards the SciTag supplied with a COPY request to the
pool via RemoteHttpDataTransferProtocolInfo. The active pool uses it for
its own firefly markers and billing record, but never sends it to the
remote (passive) party. RemoteHttpDataTransferProtocol only forwards
the TransferHeader* headers, so a client sending a plain "SciTag" header
(as FTS does) results in no SciTag on the GET/PUT/HEAD/DELETE requests
issued by the pool. The passive endpoint therefore cannot classify the
flow, and its billing records (e.g. Http/Https mover records in the kafka
stream) have an empty transferTag.

Modification:
In RemoteHttpDataTransferProtocol#addHeadersToRequest, add a "SciTag"
request header carrying the transfer tag if one is set and the client
did not already supply one explicitly via "TransferHeaderSciTag". As all
requests (including retries and redirects) are built through this
method, the remote party sees the same tag for the whole transfer.

Result:
The passive endpoint of an HTTP-TPC transfer receives the same SciTag as
the active pool, so both sides record the same transferTag in billing
and can emit matching flow markers. Transfers without a tag are
unchanged.

Testing:
- system-test with billing.format.json=true, push COPY on the HTTPS
  WebDAV door (port 2881) to itself:
     before: active pool record transferTag=65, passive pool record
          transferTag="" (door logs tagSource=none)
  after:  both records transferTag=65 (door logs tagSource=SciTag-header)

Acked-by:
Target: master, 12.0
Require-book: no
Require-notes: no
(cherry picked from commit 1a5f361)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant