• Linux
• Curl

Using Curl to access web pages

• Links: Security/Web , Linux/OpenSSL

Curl cert validation

curl -v --cert-status https://vigor.nz

Curl with selfsigned cert to ip

export CURL_IP="10.1.1.5"
export CURL_HOST="myhost.co.nz"
export CURL_CA_FILE="ca_my_selfsigned_cert.pem"

curl -i --cacert ${CURL_CA_FILE} --resolve ${CURL_HOST}:443:${CURL_IP} "https://${CURL_HOST}/"

Set headers

• Check Headers e.g. CORS (access-control-allow-origin:) headers

  $ curl -k -sD - -H "Origin: https://wiki.vigor.nz" -o /dev/null https://wiki.vigor.nz/

  • or

    curl -H "Origin: https://mydomain.nz" -H "Access-Control-Request-Method: GET" -H "Access-Control-Request-Headers: X-Requested-With"   -X OPTIONS --verbose https://vigor.nz 2>&1 | grep -i "access\|cors"

Use Curl to measure web page response

• From: https://stackoverflow.com/questions/18215389/how-do-i-measure-request-and-response-times-at-once-using-curl

• Create a new file, curl-format.txt, and paste in:

       time_namelookup:  %{time_namelookup}s\n
          time_connect:  %{time_connect}s\n
       time_appconnect:  %{time_appconnect}s\n
      time_pretransfer:  %{time_pretransfer}s\n
         time_redirect:  %{time_redirect}s\n
    time_starttransfer:  %{time_starttransfer}s\n
                       ----------\n
            time_total:  %{time_total}s\n

• Make a request:

   curl -w "@curl-format.txt" -o /dev/null -s "http://wordpress.com/"

  • Or on Windows, it's...

     curl -w "@curl-format.txt" -o NUL -s "http://wordpress.com/"

  • All in one, using ?[1-10] for repeat and add --parallel

    curl -w "%{time_namelookup}s\n time_connect:  %{time_connect}s\n time_appconnect:  %{time_appconnect}s\n time_pretransfer:  %{time_pretransfer}s\n time_redirect:  %{time_redirect}s\n time_starttransfer:  %{time_starttransfer}s\n" -o /dev/null -s "https://wiki.vigor.nz/?[1-10]" --parallel 

Curl with http proxy

• curl -x myproxy.com:8080 https://reqbin.com/echo

Curl and set useragent

• curl -i --user-agent "Googlebot" https://www.google.com/

Curl with host header/SNI

•  curl -i -H "Host: test-public.example.com" https://test.example.com/HealthCheck

Curl check SSL/TLS certs

for url in \
  https://google.com \
  https://microsoft.com \
    ; do
        host="${url#https://}"
        host="${host%%/*}"
        expires=$(echo | openssl s_client -servername "$host" -connect "$host:443" 2>/dev/null | openssl x509 -noout -enddate | cut -d= -f2)
        expires_at=$(date -j -f "%b %e %T %Y %Z" "$expires" +%s)
        days_remaining=$(( (expires_at - $(date +%s)) / 86400 ))
        printf '# %-28s %4d days (%s)\n' "$host" "$days_remaining" "$expires"
done

CategoryLinux