Namecoin Core stuck syncing using Tor

[biolizard89]

I've been running Namecoin Core inside a Qubes-Whonix VM for testing purposes. About 1.5 days ago, it stopped downloading blocks. At the moment, it seems to be oscillating between 0 and 1 connections to the Namecoin network, and the block count isn't increasing (it's stuck at 240293). I'm not sure if this is a Tor problem or a Namecoin problem. Namecoin Core is built from latest GitHub source code (the About screen shows it as version 3b70b51). SOCKS proxy is set to 10.152.152.10:9111 (this is the Whonix SOCKS port designated for Bitcoin), although disabling the SOCKS proxy (which uses Whonix's Trans port) doesn't help. I have a NamecoinQ installation (built from latest GitHub source code) in another Qubes-Whonix VM, which is having no trouble syncing.

debug.log:

Code: Select all

2015-07-23 12:35:01 Namecoin version v0.11.99.0-3b70b51 (2015-07-21 07:59:44 +0200)
2015-07-23 12:35:01 Using OpenSSL version OpenSSL 1.0.1e 11 Feb 2013
2015-07-23 12:35:01 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2015-07-23 12:35:01 Default data directory /home/user/.namecoin
2015-07-23 12:35:01 Using data directory /home/user/.namecoin
2015-07-23 12:35:01 Using config file /home/user/.namecoin/namecoin.conf
2015-07-23 12:35:01 Using at most 125 connections (1024 file descriptors available)
2015-07-23 12:35:01 Using 8 threads for script verification
2015-07-23 12:35:01 scheduler thread start
2015-07-23 12:35:01 Binding RPC on address ::1 port 8336 (IPv4+IPv6 bind any: 0)
2015-07-23 12:35:01 Binding RPC on address 127.0.0.1 port 8336 (IPv4+IPv6 bind any: 0)
2015-07-23 12:35:01 Using wallet wallet.dat
2015-07-23 12:35:01 init message: Verifying wallet...
2015-07-23 12:35:01 CDBEnv::Open: LogDir=/home/user/.namecoin/database ErrorFile=/home/user/.namecoin/db.log
2015-07-23 12:35:01 Cache configuration:
2015-07-23 12:35:01 * Using 2.0MiB for block index database
2015-07-23 12:35:01 * Using 32.5MiB for chain state database
2015-07-23 12:35:01 * Using 65.5MiB for in-memory UTXO set
2015-07-23 12:35:01 init message: Loading block index...
2015-07-23 12:35:01 Opening LevelDB in /home/user/.namecoin/blocks/index
2015-07-23 12:35:01 Opened LevelDB successfully
2015-07-23 12:35:01 Opening LevelDB in /home/user/.namecoin/chainstate
2015-07-23 12:35:02 Opened LevelDB successfully
2015-07-23 12:35:03 LoadBlockIndexDB: last block file = 14
2015-07-23 12:35:03 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=11603, size=108341007, heights=228691...240293, time=2015-04-30...2015-07-21)
2015-07-23 12:35:03 Checking all blk files are present...
2015-07-23 12:35:03 LoadBlockIndexDB: transaction index disabled
2015-07-23 12:35:03 LoadBlockIndexDB(): name history disabled
2015-07-23 12:35:03 LoadBlockIndexDB: hashBestChain=b1a169de56c94ceb40a8d0d092566b87fa145572c95eeff0e6b553f8bffb3cf0 height=240293 date=2015-07-21 14:23:14 progress=0.998151
2015-07-23 12:35:03 init message: Verifying blocks...
2015-07-23 12:35:03 Verifying last 288 blocks at level 3
2015-07-23 12:35:04 No coin database inconsistencies in last 289 blocks (4370 transactions)
2015-07-23 12:35:04  block index            3377ms
2015-07-23 12:35:04 init message: Loading wallet...
2015-07-23 12:35:04 nFileVersion = 119900
2015-07-23 12:35:04 Keys: 101 plaintext, 0 encrypted, 101 w/ metadata, 101 total
2015-07-23 12:35:04  wallet                  380ms
2015-07-23 12:35:04 init message: Activating best chain...
2015-07-23 12:35:04 mapBlockIndex.size() = 240580
2015-07-23 12:35:04 nBestHeight = 240293
2015-07-23 12:35:04 setKeyPool.size() = 100
2015-07-23 12:35:04 mapWallet.size() = 0
2015-07-23 12:35:04 mapAddressBook.size() = 1
2015-07-23 12:35:04 init message: Loading addresses...
2015-07-23 12:35:04 ERROR: Read: Failed to open file /home/user/.namecoin/banlist.dat
2015-07-23 12:35:04 Invalid or missing banlist.dat; recreating
2015-07-23 12:35:04 Loaded 2 addresses from peers.dat  2ms
2015-07-23 12:35:04 dnsseed thread start
2015-07-23 12:35:04 net thread start
2015-07-23 12:35:04 msghand thread start
2015-07-23 12:35:04 addcon thread start
2015-07-23 12:35:04 init message: Done loading
2015-07-23 12:35:04 opencon thread start
2015-07-23 12:35:05 GUI: PaymentServer::LoadRootCAs: Loaded  169  root certificates 
2015-07-23 12:35:05 SOCKS5 connecting 79.134.235.130
2015-07-23 12:35:15 Loading addresses from DNS seeds (could take a while)
2015-07-23 12:35:15 0 addresses found from DNS seeds
2015-07-23 12:35:15 dnsseed thread exit
2015-07-23 12:35:25 ERROR: Error reading proxy response
2015-07-23 12:35:25 SOCKS5 connecting nmc.seed.quisquis.de
2015-07-23 12:35:26 SOCKS5 connected nmc.seed.quisquis.de
2015-07-23 12:35:26 SOCKS5 connecting 116.14.28.167
2015-07-23 12:35:26 receive version message: : version 38000, blocks=240579, us=77.247.181.162:52656, peer=1
2015-07-23 12:35:26 Added time data, samples 2, offset +3 (+0 minutes)
2015-07-23 12:35:27 SOCKS5 connected 116.14.28.167
2015-07-23 12:35:27 SOCKS5 connecting namecoindnsseed.digi-masters.com
2015-07-23 12:35:28 SOCKS5 connected namecoindnsseed.digi-masters.com
2015-07-23 12:35:28 receive version message: : version 37602, blocks=240579, us=96.44.189.100:25157, peer=3
2015-07-23 12:35:28 SOCKS5 connecting namecoindnsseed.digi-masters.uk
2015-07-23 12:35:48 ERROR: Error reading proxy response
2015-07-23 12:35:50 SOCKS5 connecting dnsseed.namecoin.webbtc.com
2015-07-23 12:35:51 SOCKS5 connected dnsseed.namecoin.webbtc.com
2015-07-23 12:35:51 receive version message: /bitcoinruby:0.0.6/: version 70001, blocks=240579, us=144.76.183.77:8334, peer=4
2015-07-23 12:35:52 SOCKS5 connecting namecoindnsseed.digi-masters.uk
2015-07-23 12:35:57 SOCKS5 connected namecoindnsseed.digi-masters.uk
2015-07-23 12:35:57 receive version message: : version 38000, blocks=240579, us=37.187.129.166:60072, peer=5
2015-07-23 12:36:38 SOCKS5 connecting 79.134.235.130
2015-07-23 12:36:58 ERROR: Error reading proxy response
2015-07-23 12:37:10 SOCKS5 connecting 116.14.28.167
2015-07-23 12:37:12 SOCKS5 connected 116.14.28.167
2015-07-23 12:38:24 SOCKS5 connecting 116.14.28.167
2015-07-23 12:38:29 SOCKS5 connected 116.14.28.167
2015-07-23 12:39:40 SOCKS5 connecting 79.134.235.130
2015-07-23 12:40:00 ERROR: Error reading proxy response
2015-07-23 12:40:12 SOCKS5 connecting 79.134.235.130
2015-07-23 12:40:32 ERROR: Error reading proxy response
2015-07-23 12:40:44 SOCKS5 connecting 79.134.235.130
2015-07-23 12:40:54 ERROR: Proxy error: TTL expired
2015-07-23 12:41:05 SOCKS5 connecting 116.14.28.167
2015-07-23 12:41:07 SOCKS5 connected 116.14.28.167
2015-07-23 12:42:19 SOCKS5 connecting 116.14.28.167
2015-07-23 12:42:22 SOCKS5 connected 116.14.28.167
2015-07-23 12:43:35 SOCKS5 connecting 116.14.28.167
2015-07-23 12:43:37 SOCKS5 connected 116.14.28.167
2015-07-23 12:44:49 SOCKS5 connecting 79.134.235.130
2015-07-23 12:45:09 ERROR: Error reading proxy response
2015-07-23 12:45:21 SOCKS5 connecting 116.14.28.167
2015-07-23 12:45:24 SOCKS5 connected 116.14.28.167
2015-07-23 12:46:36 SOCKS5 connecting 79.134.235.130
2015-07-23 12:46:56 ERROR: Error reading proxy response
2015-07-23 12:47:07 SOCKS5 connecting 79.134.235.130
2015-07-23 12:47:27 ERROR: Error reading proxy response
2015-07-23 12:47:39 SOCKS5 connecting 79.134.235.130
2015-07-23 12:47:59 ERROR: Error reading proxy response
2015-07-23 12:48:11 SOCKS5 connecting 116.14.28.167
2015-07-23 12:48:15 SOCKS5 connected 116.14.28.167

Any ideas what might be causing this problem?

[cassini]

/bitcoinruby:0.0.6/: version 70001, blocks=240579, us=144.76.183.77:8334, peer=4

It seems this has nothing to do with Tor. First of all, NamecoinCore clients and NamecoinRuby-based clients are completely incompatible, see
https://forum.namecoin.info/viewtopic.p ... 103#p15103
NamecoinCore sends a getheaders message, and NamecoinRuby ignores it and sends several getblocks messages instead, and then silence.

Secondly, NamecoinCore also has difficulties to actively connect to Namecoin-Qt 0.3.80 clients and older. The other way round is no problem: starting a 0.3.80 with the -connect argument connecting to a NamecoinCore client works instantly. I think it is again this initial getheaders message the older clients have problems with.

[biolizard89]

/bitcoinruby:0.0.6/: version 70001, blocks=240579, us=144.76.183.77:8334, peer=4

It seems this has nothing to do with Tor. First of all, NamecoinCore clients and NamecoinRuby-based clients are completely incompatible, see
https://forum.namecoin.info/viewtopic.p ... 103#p15103
NamecoinCore sends a getheaders message, and NamecoinRuby ignores it and sends several getblocks messages instead, and then silence.

Interesting. I'm actually not familiar with the NamecoinRuby implementation. A quick Disconnect shows the repo to be at https://github.com/mhanne/namecoin-ruby . I'll add it to my list of Namecoin implementations (to be added to namecoin.org soon). Is this interoperability bug a problem in NamecoinRuby specifically, or does this also affect BitcoinRuby and Bitcoin Core?

Secondly, NamecoinCore also has difficulties to actively connect to Namecoin-Qt 0.3.80 clients and older. The other way round is no problem: starting a 0.3.80 with the -connect argument connecting to a NamecoinCore client works instantly. I think it is again this initial getheaders message the older clients have problems with.

That makes sense. So does that imply that my problem would be solved if there were more Namecoin Core nodes on the Internet that accepted incoming connections? I think this is related to Tor as well, in the sense that being behind Tor means I can't receive incoming connections from the public Internet, so 0.3.x clients can't make connections to me (which would solve the problem, presumably).

[cassini]

or does this also affect BitcoinRuby and Bitcoin Core?

This is what I want to find out next. I've already searched the Bitcoin Core github history if they had a similar issue with BitcoinRuby. Couldn't find anything. Maybe I should just try connecting BitcoinCore with a BitcoinRuby client and watch the initial communication between those two.

So does that imply that my problem would be solved if there were more Namecoin Core nodes on the Internet that accepted incoming connections?

Yes, I guess so.

I think this is related to Tor as well, in the sense that being behind Tor means I can't receive incoming connections from the public Internet, so 0.3.x clients can't make connections to me

ok, looking at it this way I agree.

[biolizard89]

I recently rebuilt Namecoin Core from latest GitHub source code (Namecoin Core version v0.11.99.0-527303e (64-bit)) and tried it out on Whonix 11 (was using Whonix 10 when I originally made this thread). The profile folder was copied from a working Fedora 21 VM (with the original intent of not needing to redownload the whole chain). Now it seems to work fine, it quickly finds 8 peers when using the SOCKS port. I'm not certain if this is due to a Namecoin Core bugfix, a Whonix bugfix, having more Namecoin Core nodes on the network, or just a side effect of copying a working profile folder in. I'll do some testing later to see if it's really fixed.

[domob]

I recently rebuilt Namecoin Core from latest GitHub source code (Namecoin Core version v0.11.99.0-527303e (64-bit)) and tried it out on Whonix 11 (was using Whonix 10 when I originally made this thread). The profile folder was copied from a working Fedora 21 VM (with the original intent of not needing to redownload the whole chain). Now it seems to work fine, it quickly finds 8 peers when using the SOCKS port. I'm not certain if this is due to a Namecoin Core bugfix, a Whonix bugfix, having more Namecoin Core nodes on the network, or just a side effect of copying a working profile folder in. I'll do some testing later to see if it's really fixed.

As far as I know, peer discovery doesn't work at the moment purely over Tor. This is because DNS seeds seemingly do not work and we do not yet have an extracted, hardcoded list of "fallback nodes" including .onion ones. But if you -addnode some known nodes or copy a working profile folder which already has a set of peers, it works.

[biolizard89]

I recently rebuilt Namecoin Core from latest GitHub source code (Namecoin Core version v0.11.99.0-527303e (64-bit)) and tried it out on Whonix 11 (was using Whonix 10 when I originally made this thread). The profile folder was copied from a working Fedora 21 VM (with the original intent of not needing to redownload the whole chain). Now it seems to work fine, it quickly finds 8 peers when using the SOCKS port. I'm not certain if this is due to a Namecoin Core bugfix, a Whonix bugfix, having more Namecoin Core nodes on the network, or just a side effect of copying a working profile folder in. I'll do some testing later to see if it's really fixed.

As far as I know, peer discovery doesn't work at the moment purely over Tor. This is because DNS seeds seemingly do not work and we do not yet have an extracted, hardcoded list of "fallback nodes" including .onion ones. But if you -addnode some known nodes or copy a working profile folder which already has a set of peers, it works.

Ah, I see. So that means that what I did is actually an anonymity-breaker too, since my peer list was preserved from a profile that connected over clearnet. The per-peer stream isolation introduced in 0.11 definitely makes this less of an issue, though. Anonymity is complicated....

Do DNS seeds not work because they're over UDP (which Tor generally can't tunnel)? Would Whonix's transparent DNS routing be likely to help with this? My previous tests were with a SOCKS port enabled. While a properly designed Bitcoin client could access DNS over Tor's SOCKS port, I have no idea if Bitcoin Core supports this, nor do I know what bad things could happen as a result if a Tor exit hijacked those DNS requests.

[domob]

Do DNS seeds not work because they're over UDP (which Tor generally can't tunnel)? Would Whonix's transparent DNS routing be likely to help with this? My previous tests were with a SOCKS port enabled. While a properly designed Bitcoin client could access DNS over Tor's SOCKS port, I have no idea if Bitcoin Core supports this, nor do I know what bad things could happen as a result if a Tor exit hijacked those DNS requests.

To be honest, I have no idea. I guess it is broken because -proxy does not work with UDP in Bitcoin, but it may be due to something else. In any case, this is presumably the same upstream - so you can try to file an issue against Bitcoin to see what they tell you.

[biolizard89]

Do DNS seeds not work because they're over UDP (which Tor generally can't tunnel)? Would Whonix's transparent DNS routing be likely to help with this? My previous tests were with a SOCKS port enabled. While a properly designed Bitcoin client could access DNS over Tor's SOCKS port, I have no idea if Bitcoin Core supports this, nor do I know what bad things could happen as a result if a Tor exit hijacked those DNS requests.

To be honest, I have no idea. I guess it is broken because -proxy does not work with UDP in Bitcoin, but it may be due to something else. In any case, this is presumably the same upstream - so you can try to file an issue against Bitcoin to see what they tell you.

Okay. Yeah, if it's because Bitcoin Core can't proxy DNS over UDP, I believe using Whonix's transparent routing will work. However, that's not a real solution since it doesn't support stream isolation. At least it will be a useful diagnostic test. I'll do the experiment when I have a few minutes. Once I have more info, I'll look into filing an issue with Bitcoin Core.