## Purpose
-An encoding/decoding method that allows users to compress their human-readable data into fewer humand-readable characters than other popular notation systems, for any arbitrary underlying machine-readable binary string.
+An encoding/decoding method that allows users to compress their human-readable data into fewer human-readable characters than other popular notation systems, for any arbitrary underlying machine-readable binary string.
## Warning:
-This software is still in its experimental phase and should not be relied upon for production.
+This software is still in its experimental phase and should not be relied upon for production.
## Background on Mnemonics (private keys) and Human vs Machine-readable code
-Mnemonics (aka recovery phrases) are used in many popular crypto wallet applciations including BIP39 (which follows a specific wordlist and checksum requirement, among other steps for wallet derivation such as BIP32, BIP44) enable a user to backup their initial entropy in human-readable format.
+Mnemonics (aka recovery phrases) are used in many popular crypto wallet applciations including [BIP39 (https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki) (which follows a specific wordlist and checksum requirement, among other steps for wallet derivation such as BIP32, BIP44) enable a user to backup their initial entropy in human-readable format.
For example, instead of a user having to backup a string of 128 bits or their private key, they can simply store the encoded mnemonic which represents those bits or a private key.
@@ -32,7 +32,7 @@ For example, instead of a user having to backup a string of 128 bits or their pr
|Total Words |`132/11 = 12 words` |`264/11 = 24 words` |
|----------------|-------------------------------|-----------------------------|
-In terms of actual pre-image resistance, the initial entropy should be generated in a cryptographically-secure manner that is pre-image resistant and resistant to other attacks, such as outlined in the W3C Cryptography API or via the "secrets" module in Python), and the psuedo-random binary string that results will be that is machine-readable where the purpose of encoding it into a mnemonic is to make it easier to notate, recite, read, and write, compared to binary (machine-readable).
+In terms of actual pre-image resistance, the initial entropy should be generated in a cryptographically-secure manner that is pre-image resistant and resistant to other attacks, such as outlined in the [W3C Cryptography API](https://www.w3.org/TR/WebCryptoAPI/) or via the [secrets](https://docs.python.org/3/library/secrets.html) module in Python), and the psuedo-random binary string that results will be machine-readable where the purpose of encoding it into a mnemonic is to make it easier to notate, recite, read, and write, (i.e. human-readable) compared to binary (machine-readable).
@@ -62,15 +62,18 @@ The Hatnotation system is *not intended to be an alternative to human-readable m
Just as a mnemonic that represents a 132 bits of some initial entropy should convey 128 bits of security if generated properly, as the last 4 bits are deterministically derived from the hash-based checksum computation (hashing the initial entropy as a byte array). Those same 132 bits can be encoded using the Hatnotation system which will result in 22 characters, sourced from the 64 character library. Mathematically, 64^22 == 2^132, hence why there is no information/security loss, for example.
### Lemma
-For any arbitrary binary (base-2) string x of length n, after hatnotation is applied to x, the length n = ((x - (x % 6))/6)+(x %6, optionally in some cases depending on how the encoder/decoder is constructed). Optimal compression will occur when the [COMPLETE THIS SECTION]
+For any arbitrary binary (base-2) string x of length n, after hatnotation is applied to x, the length n = ((x - (x % 6))/6)+(x %6, optionally in some cases depending on how the encoder/decoder is constructed).
-- `(128-(128 % 6)) / 6 == 21 == 2^128`
-- `(132-(132 % 6)) / 6 == 22 == 2^132`
-- `(256-(256 % 6)) / 6 == 42 == 2^256`
-- `(264-(264 % 6)) / 6 == 44 == 2^264`
+Optimal compression will occur when the x modulo 6 is equal to zero, and least optimum when x modulo 6 is equal to 5 (The assumption in this last sentence should be checked).
+Below are two examples of when notation is optimum, using 132 bits and 264 bits as an example, and which are standard bit-lengths for the underlying entropy that represents 12-word or 24-word mnemonics used with crypto wallets.
-| Private Key |12-word mnemonic |24-word mnemonic |
+- `(64^((132-(132 % 6)) / 6) == 2^132`
+
+- `(64^((264-(264 % 6)) / 6) == 2^264`
+
+
+| Private Key |Length after Hatnotation | BIP39 checksum |
|----------------|-------------------------------|-----------------------------|
| 128 bits |21 characters |Checksum must be computed |
| 132 bits |22 characters |Checksum included |
@@ -139,7 +142,7 @@ Using the binascii libary in python which contains the string library, we source
47 | "101111", | ",",
48 | "110000", | "-",
49 | "110001", | ".",
-50 | "110010", | "/",
+50 | "110010", | "{",
51 | "110011", | ":",
52 | "110100", | ";",
53 | "110101", | "<",
@@ -148,7 +151,7 @@ Using the binascii libary in python which contains the string library, we source
56 | "111000", | "?",
57 | "111001", | "@",
58 | "111010", | "[",
-59 | "111011", | "\",
+59 | "111011", | "}",
60 | "111100", | "]",
61 | "111101", | "^",
62 | "111110", | "_",
@@ -156,14 +159,17 @@ Using the binascii libary in python which contains the string library, we source
## Library verification
-In Python version 3.7 using the strings library, the following steps can be taken to verify the library and character order and by omitting the last 4 characters "{|}~" in the fourth step below (zeroed-index values 64-67 which have been omitted from the above list as well):
+In Python version 3.7 using the strings library, the following steps can be taken to verify the library and character order and notice that the following 4 characters are omitted "\\~/|" in the fourth step below:
- ```>>> import string```
- ```>>> dir(string)['Formatter', 'Template', '_ChainMap', '_TemplateMetaclass', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_re', '_string', 'ascii_letters', 'ascii_lowercase', 'ascii_uppercase', 'capwords', 'digits', 'hexdigits', 'octdigits', 'printable', 'punctuation', 'whitespace']```
- ```>>> print(string.digits+string.ascii_uppercase+string.punctuation)```
- ```0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~```
-(i.e. ~~```"{|}~"```~~)
+- Note, the backslash `\` and forwardslash `/`characters were swapped with opening `{` and closing `}` curly brackets in the following issue: https://github.com/hatgit/hatnotation/issues/3.
+- The list of valid Hatnotation library characters are thus as follows: 0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&'()*+,-.{:;<=>?@[}]^_`
+
+(and the following four remain excluded/reserved ~~```"\/|~"```~~).
## Requirements
@@ -174,8 +180,31 @@ Python 3 or higher
## Tests:
-## Reources:
+> Example Test strings:
+
+- Decode Target: `HELLOWORLD
+
+- Each letter decodes to respective 6-bit group: `"010001","001110","010101","010101","011000"," ", "100000","011000","011011","010101","001101",
+
+- Each Word as Continous string `"010001001110010101010101011000" "100000011000011011010101001101"
+
+- Concatenation of both words into one string: `"010001001110010101010101011000100000011000011011010101001101"
+
+- Converted binary string to hex (can be used as starting point to encode to "helloworld": `0x44e55562061b54d`
+
+
+Second Test: Decoder should be able to take a single backslash `\` and return index value 59, as well as multiple ones '`\\\\\`' and not be affected by any formatting/rendering issues of the app or interface (i.e. python or javascript/html).
+
+
+## Resources:
+
+- Other base64 encoding schemes: https://en.wikipedia.org/wiki/Base64
+- Notational conventions: https://tools.ietf.org/html/rfc2822
+- Human-readable keys (1994) https://tools.ietf.org/html/rfc1751
## Roadmap/Plans:
+- develop a range of potential use cases
+- potentially propose a request for comments (RFC) for consideration as a standard.
+- add error message for invalid characters (i.e. lowercase and reserved characters `\|/~`
diff --git a/base64Decode_v1_01.py b/base64Decode_v1_01.py
new file mode 100644
index 0000000..16ee090
--- /dev/null
+++ b/base64Decode_v1_01.py
@@ -0,0 +1,114 @@
+import string
+import binascii
+import math
+import secrets
+
+#Author: Steven Hatzakis @2018, Licensed under Apache 2.0
+
+#Version 1.01
+
+
+# Example Test string target to decode: HELLOWORLD
+# Each letter decodes to respective 6-bit group: `"010001","001110","010101","010101","011000"," ", "100000","011000","011011","010101","001101",
+# Each Word as Continous string `"010001001110010101010101011000" "100000011000011011010101001101"
+# Concatenation of both words into one string: `"010001001110010101010101011000100000011000011011010101001101"
+# Converted binary string to hex: `0x44e55562061b54d`
+
+
+b64dict=dict()
+
+base64library=["0",
+"1",
+"2",
+"3",
+"4",
+"5",
+"6",
+"7",
+"8",
+"9",
+"A",
+"B",
+"C",
+"D",
+"E",
+"F",
+"G",
+"H",
+"I",
+"J",
+"K",
+"L",
+"M",
+"N",
+"O",
+"P",
+"Q",
+"R",
+"S",
+"T",
+"U",
+"V",
+"W",
+"X",
+"Y",
+"Z",
+"!",
+'"',
+"#",
+"$",
+"%",
+"&",
+"'",
+"(",
+")",
+"*",
+"+",
+",",
+"-",
+".",
+"}",
+":",
+";",
+"<",
+"=",
+">",
+"?",
+"@",
+"[",
+"{",
+"]",
+"^",
+"_",
+"`"]
+for i in range(len(base64library)):
+ b64dict[base64library[i]] = i
+
+
+def decodeAsBin(string):
+ output = []
+ s=''
+ print(len(string))
+ for j in range(len(string)):
+
+ if (string[j] != ' '):
+ output.append(bin(b64dict[string[j].upper()])[2:].zfill(6))
+
+ return s.join(output)
+
+
+def decodeAsHex(string):
+
+ return hex(int(decodeAsBin(string), 2))
+
+hatdata2decode=input('Enter any combination of the following characters to decode without spaces: "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&''()*+,-.{:;<=>?@[}]^_`":')
+
+print(decodeAsBin(hatdata2decode))
+
+## test decode string "HELLOWORLD" should return the binary equivalent of hex: 0x44e55562061b54d
+## test decode string "INVENTEDBYSTEVENHATZAKIS@2018' should return this hex 0x125df39774e34b89c74e7ce5d129d8ca512739080048 as equivalent in binary to this output: 010010010111011111001110010111011101001110001101001011100010011100011101001110011111001110010111010001001010011101100011001010010100010010011100111001000010000000000001001000
+
+## Note/CONSIDERATION: In version 1.01 to resolve a rendering conflict, the Forward slash `/` and Backslash `\` characters have been swapped out and replaced by the closing curly bracket `}` and opening curly bracket `{` on purpose.
+
+
+
diff --git a/base64encode_v1_01.py b/base64encode_v1_01.py
new file mode 100644
index 0000000..3363b34
--- /dev/null
+++ b/base64encode_v1_01.py
@@ -0,0 +1,104 @@
+import math # required
+import secrets ## optional for future use
+import string # required
+import binascii # required
+
+#Author: Steven Hatzakis @2018, Licensed under Apache 2.0
+# Version 1.01
+
+#test binary data string: 0b010010010111011111001110010111011101001110001101001011100010011100011101001110011111001110010111010001001010011101100011001010010100010010011100111001000010000000000001001000
+
+#test string as hex: '0x125df39774e34b89c74e7ce5d129d8ca512739080048'
+
+#test string result :'INVENTEDBYSTEVENHATZAKIS@2018'
+
+## legacy notes: #FIX: LEADING ZEROS LOST (i.e.w/ test string 0x064ce8c835f4d374e04e33244beccad0)
+#also, even when no leading zero in string, the leading zeroes of the actual base64 character are discarded,
+#as the encoding appears to happen from right to left, see this test string where leading zeroes of
+# first char 2 are discarded, and last char is N 0xb13ae7e331ce9dfa59799e95ee8dc117
+
+def base64en(address_hex): # important never to rename base64en to "base64" which can corrupt the Python installation.
+ # (renamed alphabet to base64library below) alphabet = string.digits+string.ascii_uppercase+string.punctuation
+ base64library=["0",
+"1",
+"2",
+"3",
+"4",
+"5",
+"6",
+"7",
+"8",
+"9",
+"A",
+"B",
+"C",
+"D",
+"E",
+"F",
+"G",
+"H",
+"I",
+"J",
+"K",
+"L",
+"M",
+"N",
+"O",
+"P",
+"Q",
+"R",
+"S",
+"T",
+"U",
+"V",
+"W",
+"X",
+"Y",
+"Z",
+"!",
+'"',
+"#",
+"$",
+"%",
+"&",
+"'",
+"(",
+")",
+"*",
+"+",
+",",
+"-",
+".",
+"{",
+":",
+";",
+"<",
+"=",
+">",
+"?",
+"@",
+"[",
+"}",
+"]",
+"^",
+"_",
+"`"]
+ b64_string = ''
+ # Get the number of leading zeros and convert hex to decimal
+ leading_zeros = len(str(address_hex))-2 #len(str(address_hex)) - len(str(address_hex.lstrip('0')))
+ # Convert hex to decimal
+ address_int = int(address_hex)
+ # Append digits to the start of string
+ while address_int > 0:
+ digit = address_int % 64
+ digit_char = base64library[digit]
+ b64_string = digit_char + b64_string
+ address_int //= 64
+ # Add '1' for each 2 leading zeros
+ #ones = leading_zeros // 8
+ #for one in range(ones):
+ # b64_string = '1' + b64_string
+ return b64_string
+hexer=int(input('enter hex with pad'),16)
+print(base64en(hexer))
+print("The above is this many char's long: ",len(str((base64en(hexer)))))