CAT and ftp

[Ratsima]

I wonder if anyone else is having trouble uploading files via ftp over CAT. My Cha Am webcam uses curl (ftp) to upload images to my web host. I also have a webcam in Korat which uploads to the same host using curl, but over TOT. The TOT uploads almost never fail, but the CAT uploads fail regularly, usually with curl error code 28 (timeout) or 56 (network data failure).

I've done everything I can think of to overcome this error including increasing the timeout delay, making multiple retries for each image and disabling extended passive mode. Nothing works.

The Cha Am router is run by the condo, so there's no way for me to check for blocked ports. But, I figure if it were blocked ports it would always fail. It doesn't.

Any ideas?

[buksida]

Tried a different FTP client such as Filezilla or using a VPN?

This should at least determine whether the problem is ISP or server related.

[Ratsima]

Well, I've tried both curl and ftp from the command line and get the same sort of failures. The only GUI client I've tried is Cyberduck. Same result. Sometimes it works and sometimes it doesn't.

I know it's not the server, because uploads work fine from Korat.

Here's the workaround I'm using now:

• Use scp to copy the file from Cha Am to Korat
  Use launchd to watch for the file in Korat
  When it arrives in Korat from Cha Am, use curl to upload it to the host

Awkward, but it works.

[hhinner]

Might be being technically naïve here, but could it be a difference between the two networks and related to whether using active or passive FTP? Maybe you might ask CAT of they are perhaps double NATing or the network is otherwise configured to interfere with FTP. Just musing, haven't really thought it out.

[Ratsima]

Yes, double NAT is involved. My condo has a single public routable IP address. They give each unit a single private IP address. I have a router which distributes private IP addresses on a completely different subnet. This works well for everything. I can even remotely log on to the Cha Am network from Korat using Chrome Remote Desktop. And, ftp works about 70% of the time.

Of course, I have to use passive ftp. My guess is that sometimes the host picks a passive ftp port that is either blocked or in use by someone else.

The odd thing is that I can go weeks with no errors at all and then experience dozens of errors in a single day.

Any way to fix this or does the double NAT insure problems?

[hhinner]

Why do you have to use passive? Does the server not allow active? Or is it because of your own firewall settings? If you can use active mode it could then be more like your remote desktop connection, always using the same port for FTP data.

[Ratsima]

I will try....

[hhinner]

Good luck!

[Ratsima]

Here's the frustration of troubleshooting this. Today, both active and passive mode are working equally well….

Code: Select all

ftp> passive
Passive mode: on; fallback to active mode: on.
ftp> put test.jpg
local: test.jpg remote: test.jpg
229 Extended Passive mode OK (|||28932|)
150 Accepted data connection
100% |***********************************| 24041       60.97 MiB/s    00:00 ETA
226-File successfully transferred
226 0.391 seconds (measured here), 60.04 Kbytes per second
24041 bytes sent in 00:00 (59.87 KiB/s)
ftp> passive
Passive mode: off; fallback to active mode: off.
ftp> put test.jpg
local: test.jpg remote: test.jpg
200 PORT command successful
150 Connecting to port 64736
100% |***********************************| 24041      142.40 MiB/s    00:00 ETA
226-File successfully transferred
226 0.393 seconds (measured here), 59.72 Kbytes per second
24041 bytes sent in 00:00 (59.50 KiB/s)

Curl also works equally well in active or passive mode:

Code: Select all

Axe:webcam mnewman$ curl -u $user:$passwd -T "$FILE" ftp://$host -P -       
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 24041    0     0  100 24041      0  27415 --:--:-- --:--:-- --:--:-- 27444
Axe:webcam mnewman$ curl -u $user:$passwd -T "$FILE" ftp://$host     
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 24041    0     0  100 24041      0  13776  0:00:01  0:00:01 --:--:-- 13784

I'll keep checking, but right now my double transfer kludge seems to be working OK.

[hhinner]

Well I'll be interested to know how it resolves. It's the sort of thing that in business has to be resolved, but for private purposes a workaround may be the easiest and cheapest solution.

[Ratsima]

What I'll do is write a script that tries a periodic upload using curl in both passive and active mode and log the results. That way I'll be able to compare easily.

I first set up this webcam last December. I didn't start having curl/ftp problems until June. I wasn't even there when it started failing. Obviously, somebody somewhere did something. Mystery.

[Ratsima]

Ah, now here's a twist. Just got back from a long cycling session only to find that the Cha Am computer went down just before 10:00 AM this morning and is still unreachable. Oh, well.

I'll be there next Thursday to see what's up.

[Ratsima]

hhinner, thanks for your suggestion to try active ftp.

I've written a script that checks both active and passive ftp (via curl). The initial result is interesting:

Code: Select all

Sat Aug 13 15:23:17 ICT 2016 CurlTest Script Begin
Sat Aug 13 15:23:17 ICT 2016 passive upload start
Sat Aug 13 15:23:21 ICT 2016 passive upload upload finish
Sat Aug 13 15:23:21 ICT 2016 passive curl return 0
Sat Aug 13 15:23:26 ICT 2016 active upload start
Sat Aug 13 15:23:27 ICT 2016 active upload upload finish
Sat Aug 13 15:23:27 ICT 2016 active curl return 0

Sat Aug 13 17:05:10 ICT 2016 CurlTest Script Begin
Sat Aug 13 17:05:10 ICT 2016 passive upload start
Sat Aug 13 17:05:30 ICT 2016 passive upload upload finish
Sat Aug 13 17:05:30 ICT 2016 passive curl return 28
Sat Aug 13 17:05:35 ICT 2016 active upload start
Sat Aug 13 17:05:38 ICT 2016 active upload upload finish
Sat Aug 13 17:05:38 ICT 2016 active curl return 0

I've set it to run every couple of hours. I'll it go for a few days and check the results. (I have the uploads time out after 20 seconds. In my experience, it either works immediately, or it never works.)

Thanks again.

[Ratsima]

Just to put a cap on this, I switched to using active ftp (via curl) and all is well. Not a single upload failure since I changed. Thanks again for the help.

[hhinner]

Hope it stays that way.