BIP Draft: Segregated Data - #2222
Conversation
There was a problem hiding this comment.
Hi @MrHash, thank you for your submission.
-
Looking through the discussion at https://delvingbitcoin.org/t/bip-draft-segregated-data-a-prunable-script-isolated-block-region-for-data-carriage/2641/40, the feedback was extensive and entirely negative.
-
The first line of the README in this repository states: "People wishing to submit a BIP should first describe their idea to the bitcoindev@googlegroups.com mailing list to gather feedback on viability and community interest before working on a formal description." BIP3 also describes this in more detail. Please see the README and BIP 3 for the process.
-
In your PR description, you write, "Attempted to post for discussion on mailing list without success." When did you send your post? Sometimes it can take a fortnight or more for the mods to get through the queue and patience may be needed.
In summary, with negative community feedback in Delving, and with no discussion on the mailing list, this PR was opened prematurely, and it ought to see positive conceptual feedback from the community before opening a PR here. Let's close this for now.
|
@jonatack I attempted to start a discussion at the mailing list over 1 month ago but my submission was denied and i don't know why. I reached out to @RubenSomsen directly on socials to find out why but unfortunately didn't get a response. Instead I created a conversation on Delving which seemed to be the only option at that point. There were several misunderstandings which were not followed on after I clarified. In the end, the useful technical feedback was fully taken on board and the BIPs have been updated in the last weeks into their current form which resolve the technical issues that were raised. Finally @murchandamus suggested I make the PR which I am now doing. Negative feedback seems to have been based on either opinion on economic principle or just plain misunderstanding, partly due to imprecision in explaination by me. I also had some brief conversations with @adam3us on X where he gave some technical suggestions and feedback which I have incoporated. I've tried to follow the procedure but i've faced obstacles, and again here. I'd appreciate if you would re-open the PR for consideration properly, otherwise clear the barriers for me to post to the mailing list. In any case, negative opinion is not grounds alone for closing a technical proposal like this. thanks |
|
@MrHash the feedback on Delving was universally negative, and your reviewers there don't have unlimited time to iterate on each of your responses. I'll ask another mod about the mailing list submission. |
|
Murch suggested that I make the PR so that's what I did, because I had no idea if trying to send to the mailing list would be blocked again. I'm not trying to bother devs at all. The negativity on Delving revolved around a strange misconception that this allowed data carriage for free, which is 100% wrong. As I said, technical feedback was appreciated and the proposal fixed. Meanwhile i've been having entirely positive feedback from Adam Back on X. |
|
I took the time just now to reach out to the ML mods. One confirmed that your submission was not approved, more or less for not making sense and wasting the community's time. Another mod replied that the discussion is already ongoing on delving, so it's better to continue the conversation there.
I would suggest continuing the existing discussion on Delving, with a link to the updated/fixed version on your GitHub. That said, none of the feedback I am seeing from any of the people involved is encouraging. Good luck. |
|
From what it sounds like you are saying that if i try to submit a second time to the mailing list with the update proposal in order to reach a broader audience that it will be censored again? |
|
@MrHash The mailing list tries for a balance in order to keep it useful and readable for experienced reviewers who have limited time. That review is the bottleneck, and you are asking for it repeatedly despite negative technical feedback from all of them so far. This has been a recurring problematic pattern in submission to the list and here. I've already suggested a way forward in my previous post. |
|
@jonatack To be clear, i am not trying to waste the community's time. I'm trying to present a valid technical proposal which has been misunderstood at its surface. I have been instructed to make a PR and I attach the proof of proposed next steps. 
I also attach a screenshot of Dr Back promoting the proposal after review: 
There is more detailed proposal discussion here: https://x.com/adam3us/status/2075619857714856354 and https://x.com/adam3us/status/2071164137569145207 Given that clearly you can see there is significant and useful positive feedback, I would kindly ask that you stop treating this or my following of the procedure as a waste of community time, but as a serious albeit unusual and unfamiliar proposal from outside the existing Core team. |
|
@MrHash please read my previous comments, as I do not intend to repeat them, and kindly stop wasting our time. Posts on X have no bearing here. If Murch wishes to unlock and re-open this and review it again here, he is free to do so. |
|
FWIW, this updated proposal does not appear to make any more sense than the previous one that saw only negative technical feedback on Delving, and I see why the mail list mods did not accept it. The newer proposal would make data optional at every depth and keep consensus validation identical, fixing the reorg risk AJ Towns described. Nodes would only check a commitment plus the declared size against the weight limit; everything else would be policy. It would be a soft fork for something that doesn’t need consensus rules. One can already commit to arbitrary data hashes today. The data still consumes block weight even though no one is required to keep or serve it. Most nodes would ignore it, making availability unreliable. Use cases that care about permanence would stick to normal witness or OP_RETURN data. It would be adding protocol complexity for a feature most nodes will turn off and that has weak incentives to be used at all. Some of the best reviewers in the space were in that Delving discussion and I believe you received generous feedback there that is as expert as you'll see and need. |
|
Thanks @jonatack for your feedback here so far. I did indeed promise that I would take another look at the submission here, so I’m reopening this. |
|
@murchandamus Sure. My take so far is that @MrHash has been sea-lioning and handwaving away the consistently (polite but) negative feedback on Delving Bitcoin. (I made a poll here out of interest, and there are some comments below it, though it's of course not of any more value than the X screenshot above.) |
murchandamus
left a comment
murchandamus
left a comment
There was a problem hiding this comment.
This proposal is overly verbose. It could be improved by focusing on describing the proposed feature and foregoing speculation about adoption and implementation details.
The document is missing a Rationale section, and in several instances content that should appear in the Specification appears in the Glossary, Motivation, and Key Design Decisions, whereas the Specification contains unnecessary speculative content.
This proposal essentially describes an optional network service that enables participating nodes to retain and relay optional data payloads associated with transactions. Participating nodes only accept data payloads for which the sender has paid a fee corresponding to the data payload’s length. I surmise that this proposal may garner more support if it endorsed such a framing. The seeming expectation that nodes would broadly participate in this scheme seems naive and undermines the prospect of this proposal. I remain unconvinced that users participating in data embedding schemes would be convinced to adopt this scheme to get a small fee reduction in exchange for a large data availability hit compared to simply continuing use of the already established schemes.
|
|
||
| ## Abstract | ||
|
|
||
| This BIP defines how **Segregated Data (SegData)** *entries* propagate between *peers*. The consensus BIP places entries in a block-level `segdata` *region* committed via the coinbase and validates every block from its base serialisation alone, so no node needs the region to validate, at the tip or at any depth. Entry presence, integrity, and retention are therefore relay policy. This BIP specifies how entries travel bundled with their referencing transactions, how a block's region is reconstructed from held entries, how nodes serve recent entries and advertise deeper retention, and how a node opts out in part or in full. |
There was a problem hiding this comment.
Relay policy strikes me as the wrong term here. I would perhaps describe it as an optional node service.
|
|
||
| **Entry-bearing relay**: Relay that carries SegData entries, bundled with their transactions and regions, as opposed to entry-stripped relay, which carries only the standard base serialisation. | ||
|
|
||
| **Recent-block floor**: The most recent 288 blocks, within which a SegData peer MUST retain and serve every **available** entry. |
There was a problem hiding this comment.
This Glossary section seems to contain parts of the Specification.
|
|
||
| **Recent-block floor**: The most recent 288 blocks, within which a SegData peer MUST retain and serve every **available** entry. | ||
|
|
||
| **Retention policy**: A node's local choice of how much history to keep and which entries to exclude, above the mandatory recent-block floor. It determines what the node can serve, and its depth is what the coverage tier advertises. The amount and exclusion controls are defined in the consensus BIP (§Prunability, §Reference Implementation), and this BIP standardises only the coverage signal, not their format. |
There was a problem hiding this comment.
(§Prunability, §Reference Implementation)
I don’t see a section titled “Prunability”. Please use actual links to link to referenced sections. If you wish leave yourself reminders for outstanding todos, you could e.g., leave a “(TK)” in the corresponding spots.
|
|
||
| To enable prunability of SegData entries, a SegData *reference* output carries only a 4-byte marker and 32-byte hash of the entry. The transaction alone does not carry the bytes a block builder needs to populate `segdata`. Consensus never requires the region, but the network must move entries from originator to miners, and on to any node that opts to receive them. | ||
|
|
||
| SegData entries are deliberately outside transaction identity (see D2 below), which enables deduplication and per-entry retention but means the bytes do not travel with the transaction by construction. |
There was a problem hiding this comment.
Please link to referenced sections.
| To enable prunability of SegData entries, a SegData *reference* output carries only a 4-byte marker and 32-byte hash of the entry. The transaction alone does not carry the bytes a block builder needs to populate `segdata`. Consensus never requires the region, but the network must move entries from originator to miners, and on to any node that opts to receive them. | ||
|
|
||
| SegData entries are deliberately outside transaction identity (see D2 below), which enables deduplication and per-entry retention but means the bytes do not travel with the transaction by construction. | ||
|
|
There was a problem hiding this comment.
This section is describing the approach, however the motivation section should explain why a BIP is being written, how the existing situation presents a problem, and why the proposed idea resolves the issue or improves upon the current situation.
|
|
||
| SegData entries are deliberately outside transaction identity (see D2 below), which enables deduplication and per-entry retention but means the bytes do not travel with the transaction by construction. | ||
|
|
||
| ## Key design decisions |
There was a problem hiding this comment.
At first glance this section seems to contain a mix of content that should either appear in the Specification or in the Rationale sections.
| The default policy is to receive and relay the region, so that data stays available and the node can serve and retain it. A node MAY decline mempool entry relay while still relaying ordinary transactions, MAY take new blocks base-only rather than extended, and MAY skip or prune historical entries per its retention policy, whose depth its coverage tier then advertises (D8). A node that receives no entries anywhere still tracks consensus exactly, differing only in what it can serve. Entry-stripped relay is the rule for pre-activation and non-negotiated peers, and the explicit choice for any peer. | ||
|
|
||
| The full opt-out is a single participation switch (for concreteness, a Bitcoin Core `permitsegdata=0`). In this case the node advertises neither service bit, negotiates no entry relay, takes base-only blocks, and retains nothing, so it holds and serves no entries while validating fully. This is independent of `blocksonly`, which governs mempool transaction relay in general. A node running `blocksonly` with SegData still permitted continues to take and serve block regions. |
There was a problem hiding this comment.
Here and generally, given that this BIP does not describe a consensus change, this proposal would overall be more approachable, if it took care to frame prescribed behavior as applying to software projects implementing the proposal rather than sounding like it were making claims about node behavior in general.
| - **`NODE_SEGDATA`** = `(1 << 12)`: the peer relays SegData entries and serves the full `segdata` archive since activation. The guarantee is one of depth, the full range since activation, not of every individual entry. Per-entry exclusions are permitted and answered with `notfound`, and only systematic failure to serve within range forfeits the claim (§Retention-coverage advertising). A node not covering the full range MUST NOT set the bit, the BIP-159 rule for `NODE_NETWORK`. Service bits travel in `addr` relay, so syncing nodes discover archival coverage before connecting. | ||
| - **`NODE_SEGDATA_LIMITED`** = `(1 << 13)`: the peer relays SegData entries and guarantees serving `segdata` for the most recent 288 blocks (D7). The direct analogue of `NODE_NETWORK_LIMITED`, with the same 288-block window and the same reorg-serving rationale. Nodes setting `NODE_SEGDATA` SHOULD also set `NODE_SEGDATA_LIMITED`, as archival nodes signal both BIP-159 bits today. | ||
|
|
||
| Both bits imply entry-relay capability. By default a node retains SegData for the blocks it stores, so a default-configured unpruned node is archival and sets both bits. A node retaining less sets only `NODE_SEGDATA_LIMITED` and advertises any intermediate coverage (§Retention-coverage advertising). If a connection's advertised coverage states a tier lower than the bits imply, that coverage governs for the connection. Bits are discovery hints in `addr` gossip and may be stale, and the advertised coverage is the per-connection commitment. |
There was a problem hiding this comment.
Default behavior of nodes regarding optional features is an implementation decision. I would suggest describing the services without making claims about implementation details.
|
|
||
| Both bits imply entry-relay capability. By default a node retains SegData for the blocks it stores, so a default-configured unpruned node is archival and sets both bits. A node retaining less sets only `NODE_SEGDATA_LIMITED` and advertises any intermediate coverage (§Retention-coverage advertising). If a connection's advertised coverage states a tier lower than the bits imply, that coverage governs for the connection. Bits are discovery hints in `addr` gossip and may be stale, and the advertised coverage is the per-connection commitment. | ||
|
|
||
| - **`sendsegdata`**: an empty negotiation message sent between `version` and `verack`, following the [BIP-339](bip-0339.mediawiki) (`wtxidrelay`) pattern. A node sends it only to SegData peers (either bit) at a protocol version of 70017 or higher (proposed, pending assignment). Entry-bearing relay is active on a connection only when both sides have sent it. Otherwise the connection carries entry-stripped relay (D3). Sending `sendsegdata` after `verack` is a protocol violation. |
There was a problem hiding this comment.
This approach has meanwhile been specified as BIP434: Peer Feature Negotiation.
|
|
||
| In the mempool a node MAY hold a shared entry once or once per referencing transaction, the latter matching the full-attribution feerate below. Maximum entry size is relay policy. A node rejects a zero-length entry as nonstandard, since an empty entry carries no data yet still costs a reference output. This is relay policy rather than consensus because per-entry length lives in the region, not the base serialisation, so an empty entry a miner includes anyway is harmless and simply not relayed onward in the mempool. | ||
|
|
||
| For feerate, a transaction's weight includes the full `segdata` weight of every entry it references, even one shared with another mempool transaction. This over-estimates a shared entry (consensus counts it once per block) but never under-estimates, so feerate sorting, eviction, and RBF stay well-defined and conservative. `minrelaytxfee` applies to that full weight including the entry. Standardness checks the same attributed weight, so a reference-bearing transaction and the entries it pulls in are bounded by `MAX_STANDARD_TX_WEIGHT` like any other standard transaction. Block builders MAY refine with package-style accounting at template construction. |
There was a problem hiding this comment.
It would be preferable to skip over prescribing all these implementation details by just saying that the segdata counts toward the transaction’s weight as one weight per byte which would imply the described behavior in this section.
3 participants
Consensus and Peer Services BIPs for the Segregated Data proposal.
Following on from discussion on DelvingBitcoin. Attempted to post for discussion on mailing list without success.
I've spent plenty of time trying to get it precise, hopefully there are no glaring errors. Happy to answer any further technical questions here or on the mailing list if directed.