Page 1 of 3

"import" specification

Posted: Tue Oct 15, 2013 10:22 am
by phelix
http://dot-bit.org/Namespace:Domain_nam ... delegation

Should it be recursive?

Can we use "import" or should we use something new (shorter)? From the wiki it seems to be specific to domains (it is automatically going down "map" - or is it an error?). I would prefer it to be general and to work with general json entries.

Maybe we can lean on something existing like the behavior of the python "import" statement?

The ability of the current suggestion in the wiki to be able to only import specific parts looks elegant but is is really useful?


I would expect it to work like this: Update/overwrite the current dictionary at the current level in the order of appearance of import entries.

What should happen if there are several import statements on one level?

Re: "import" specification

Posted: Tue Oct 15, 2013 1:31 pm
by domob
phelix wrote:http://dot-bit.org/Namespace:Domain_nam ... delegation

Should it be recursive?

Can we use "import" or should we use something new (shorter)? From the wiki it seems to be specific to domains (it is automatically going down "map" - or is it an error?). I would prefer it to be general and to work with general json entries.

Maybe we can lean on something existing like the behavior of the python "import" statement?

The ability of the current suggestion in the wiki to be able to only import specific parts looks elegant but is is really useful?


I would expect it to work like this: Update/overwrite the current dictionary at the current level in the order of appearance of import entries.

What should happen if there are several import statements on one level?
I would suggest to add any keys present in the imported name, but have them overwritten by keys explicitly specified in the value. Multiple imports could be done using an array of names as value of "import", and having them processed in this order. I think we should make import non-recursive - both for performance/security reasons and because it is not really necessary (IMHO - but with non-recursive import the effective value size is again restricted by a finite bound; maybe we should restrict the maximum recursion depth instead?).

General JSON entries seems like what we should do. Which actual use-cases of Namecoin interpret import is up to them, I think we should just define the general behaviour of import and let "everyone" (DNS / NMControl, NameID, ...) decide for themselves whether or not to make use of this feature. (This means, I'm not in favour of specifying that *every* Namecoin value should be interpreted as if import as processed, only how importing should work if it is wanted. Like the "signer" field we recently discussed.)

Several import statements shouldn't be possible, at least I think that JSON disallows that. (Although of course the textual value could include them, but wouldn't this be invalid JSON then and thus something we should disallow?) Use name arrays instead.

Another question: Should import work only on the top-level JSON scope of a value, or for any object within the JSON value? (Like the "map" entry of a DNS name.) But I think this should be defined by the actual use-case specification for each namespace.

Re: "import" specification

Posted: Tue Oct 15, 2013 7:23 pm
by biolizard89
A few points.

1. "import" should not assume the "map" field; it should be generic JSON (so works with other namespaces than d/).

2. Fields added by "import" should be overridden by the existing fields. This is critical to allow cold storage to work.

3. Allowing partial import (only import a subset of fields) is critical to allow cold storage to work.

4. Recursive import is necessary to allow a logical name to span large numbers of raw names. If someone is willing to pay the fees for 50 names, they should be allowed to use them as subdomains instead of requiring them to be separate 2nd-level domains. It might be useful to include a limit on recursive depth, and recursive loops should be ignored.

And yes, we should get this spec figured out. Thanks for bringing it up. :)

Re: "import" specification

Posted: Sun Nov 03, 2013 5:28 pm
by agorism1
I have attempted to implement the "import" feature. It is all in python.
Import is (should be) recursive in my design.
It has not been tested much yet, but I also added the ability to leave posts on an your ID. or to leave comments on your own or other people's posts.

I wrote it for namecoind (it uses your browser as the GUI) , and will be updating for namecoinq soon.

https://rlevykyswlg7ibqi.onion.to/

Hopefully this will make namecoin's value go up.

Re: "import" specification

Posted: Sun Nov 03, 2013 8:15 pm
by biolizard89
agorism1 wrote:I have attempted to implement the "import" feature. It is all in python.
Import is (should be) recursive in my design.
It has not been tested much yet, but I also added the ability to leave posts on an your ID. or to leave comments on your own or other people's posts.

I wrote it for namecoind (it uses your browser as the GUI) , and will be updating for namecoinq soon.

https://rlevykyswlg7ibqi.onion.to/

Hopefully this will make namecoin's value go up.
Seeing as the spec is not decided on, I'm not sure why you implemented something without being involved in the spec design process. Also, my opinion is that this should be implemented within nmcontrol.

Re: "import" specification

Posted: Mon Nov 04, 2013 6:46 am
by domob
biolizard89 wrote:Seeing as the spec is not decided on, I'm not sure why you implemented something without being involved in the spec design process. Also, my opinion is that this should be implemented within nmcontrol.
+1 for nmcontrol. However, I think there will be alternative implementations, too, as not everyone might want to use nmcontrol. (Not sure about NameID, for instance.) But for them, I think it should be rather easy to implement "import" anyway (once the specification is defined precisely).

Re: "import" specification

Posted: Wed Nov 06, 2013 7:01 pm
by phelix
domob wrote: I would suggest to add any keys present in the imported name, but have them overwritten by keys explicitly specified in the value.
This seems counter intuitive to me. I would expect it to work like the python import, overwriting everything that is present by what is being imported. Any particular reason for this behaviour?
Multiple imports could be done using an array of names as value of "import", and having them processed in this order.
+1
I think we should make import non-recursive - both for performance/security reasons and because it is not really necessary (IMHO - but with non-recursive import the effective value size is again restricted by a finite bound; maybe we should restrict the maximum recursion depth instead?).
If recursive depth certainly would have to be restricted tightly to make ddos attacks less easy. 0, 1, 2 levels?
General JSON entries seems like what we should do. Which actual use-cases of Namecoin interpret import is up to them, I think we should just define the general behaviour of import and let "everyone" (DNS / NMControl, NameID, ...) decide for themselves whether or not to make use of this feature. (This means, I'm not in favour of specifying that *every* Namecoin value should be interpreted as if import as processed, only how importing should work if it is wanted. Like the "signer" field we recently discussed.)
+1
Several import statements shouldn't be possible, at least I think that JSON disallows that. (Although of course the textual value could include them, but wouldn't this be invalid JSON then and thus something we should disallow?) Use name arrays instead.
Python json only seems to use the last occurrence.
Another question: Should import work only on the top-level JSON scope of a value, or for any object within the JSON value? (Like the "map" entry of a DNS name.) But I think this should be defined by the actual use-case specification for each namespace.
Hmm... this would allow for several imports per name. Should we limit the total number of allowed imports? Or the size of the resulting object?
biolizard89 wrote: 1. "import" should not assume the "map" field; it should be generic JSON (so works with other namespaces than d/).
Check, no "map".
2. Fields added by "import" should be overridden by the existing fields. This is critical to allow cold storage to work.
As domob suggested... but I don't get it. Why would I use import but also have the field present in the actual value? Could you please elaborate?
3. Allowing partial import (only import a subset of fields) is critical to allow cold storage to work.
...
4. Recursive import is necessary to allow a logical name to span large numbers of raw names. If someone is willing to pay the fees for 50 names, they should be allowed to use them as subdomains instead of requiring them to be separate 2nd-level domains. It might be useful to include a limit on recursive depth, and recursive loops should be ignored.
Ignoring loops sure is a good idea.
agorism1 wrote:I have attempted to implement the "import" feature. It is all in python.
Import is (should be) recursive in my design.
It has not been tested much yet, but I also added the ability to leave posts on an your ID. or to leave comments on your own or other people's posts.

I wrote it for namecoind (it uses your browser as the GUI) , and will be updating for namecoinq soon.

https://rlevykyswlg7ibqi.onion.to/

Hopefully this will make namecoin's value go up.
Is the code available somewhere? For this application there should not be a difference between classic namecoind and namecoinq daemon/qt.
domob wrote:
biolizard89 wrote:Seeing as the spec is not decided on, I'm not sure why you implemented something without being involved in the spec design process. Also, my opinion is that this should be implemented within nmcontrol.
+1 for nmcontrol. However, I think there will be alternative implementations, too, as not everyone might want to use nmcontrol. (Not sure about NameID, for instance.) But for them, I think it should be rather easy to implement "import" anyway (once the specification is defined precisely).
It needs to be in NMControl. Maybe in a modular way so that the code can easily be integrated into other software.

Should we use "import" or should we use something new (shorter)? What about "imp" instead of import? It would also avoid any potential conflict with the existing "import" (e.g. the "map" stuff).

Re: "import" specification

Posted: Wed Nov 06, 2013 9:37 pm
by biolizard89
phelix wrote:
2. Fields added by "import" should be overridden by the existing fields. This is critical to allow cold storage to work.
As domob suggested... but I don't get it. Why would I use import but also have the field present in the actual value? Could you please elaborate?
You may be right that this doesn't matter as long as users are competent.
phelix wrote:
3. Allowing partial import (only import a subset of fields) is critical to allow cold storage to work.
...
Okay, here's an example:

d/wikileaks: {"fingerprint": "[WIKILEAKS-FINGERPRINT]", import: [["dd/wikileaks-ip", "ip"]] }
dd/wikileaks-ip: {"ip": "123.45.67.89"}

(I made up a syntax for import for the purposes of this example.)

d/wikileaks is held in cold storage. dd/wikileaks-ip is more easily accessible in case the feds raid the server farm and WikiLeaks needs to switch IP's. Now, what happens if the feds hijack dd/wikileaks-ip, and change its value to this: {"ip": "456.78.91.23", "fingerprint": "[MALICIOUS-FINGERPRINT]"}

Basically, we want to make sure that the malicious fingerprint doesn't override the fingerprint in cold storage. Disallowing overrides by itself isn't enough (what if the hijacked name uses the tls field instead of fingerprint?)
phelix wrote:
domob wrote:
biolizard89 wrote:Seeing as the spec is not decided on, I'm not sure why you implemented something without being involved in the spec design process. Also, my opinion is that this should be implemented within nmcontrol.
+1 for nmcontrol. However, I think there will be alternative implementations, too, as not everyone might want to use nmcontrol. (Not sure about NameID, for instance.) But for them, I think it should be rather easy to implement "import" anyway (once the specification is defined precisely).
It needs to be in NMControl. Maybe in a modular way so that the code can easily be integrated into other software.

Should we use "import" or should we use something new (shorter)? What about "imp" instead of import? It would also avoid any potential conflict with the existing "import" (e.g. the "map" stuff).
I think "import" is fine. As far as I know there aren't any widely used implementations that use map, so I wouldn't worry about backward compatibility. The spec is labeled a draft for a reason. :)

Re: "import" specification

Posted: Thu Nov 07, 2013 5:09 pm
by phelix
OK, makes sense. Imported data shall be "weaker" than existing data. Actually the current spec says so, too: "data is imported to the current context and then overridden by explicitly provided entries"

"import"/"imp"... ok, clarity over saving space. :|
Limit to one name recursion? name0:import name:1import name2
Arrays - who is stronger, first or last? I would say first is stronger but only to be compliant.
With two "import"s on the same level only the last one counts (seems to be JSON behavior)
No "map" stuff
self import is ignored

Loops shall be ignored - how precisely?

Should we limit the total number of allowed imports? Or the size of the resulting object? (imagine recursive imports on 20 levels --> 400 imports...). What about only allowing "import" on the root level?

"import": [ ["s/shareddata", "www"], ["s/shareddata", "ftp"] ]


Which actual use-cases of Namecoin interpret import is up to them, I think we should just define the general behaviour of import and let "everyone" (DNS / NMControl, NameID, ...) decide for themselves whether or not to make use of this feature. (This means, I'm not in favour of specifying that *every* Namecoin value should be interpreted as if import as processed, only how importing should work if it is wanted.
But I guess NMControl should always import...




Anything else?

Re: "import" specification

Posted: Wed May 13, 2015 12:19 am
by biolizard89
So, a few follow up questions:
  1. How do we handle cases where a namespace spec desires "import" to mean something namespace-specific? For example, the d/ spec allows having a subdomain called "import"; we don't want a conflict. We could try renaming the import operation to something more unique, e.g. "#import". Any symbol that we use should be generally unused by base64, DNS, or email addresses, to name a few; a # seems reasonable by first glance.
  2. Right now "import" assumes that the imported name's contents are placed into a dict. This make sense for most things, but what about the use case where we're importing two strings or two arrays and want to concatenate them? Define a "#concat" field as well? What would be the syntax?
  3. Hugo has some concerns about doing this on the JSON level rather than namespace/map level... I'll let him describe his concerns here.