[Collection] URL Decoding Methods

by

URL encoding “is a method to encode information in a Uniform Resource Identifier (URI)”. It is also called Percent-encoding because percentage symbols are used to encode certain reserved characters:

!#$%&'()*+,/:;=?@[]
%21%23%24%25%26%27%28%29%2A%2B%2C%2F%3A%3B%3D%3F%40%5B%5D

This article collects various ways to decode an URL encoded string. Let’s get started!

Python 2

$ alias urldecode='python -c "import sys, urllib as ul; \
    print ul.unquote_plus(sys.argv[1])"'

$ alias urlencode='python -c "import sys, urllib as ul; \
    print ul.quote_plus(sys.argv[1])"'

Source

Here’s an example usage:

$ urldecode 'q+werty%3D%2F%3B'
q werty=/;

$ urlencode 'q werty=/;'
q+werty%3D%2F%3B

Python 3

$ alias urldecode='python3 -c "import sys, urllib.parse as ul; \
    print(ul.unquote_plus(sys.argv[1]))"'

$ alias urlencode='python3 -c "import sys, urllib.parse as ul; \
    print (ul.quote_plus(sys.argv[1]))"'

Here’s an example usage:

$ urldecode 'q+werty%3D%2F%3B'
q werty=/;

$ urlencode 'q werty=/;'
q+werty%3D%2F%3B

Source

sed

$ sed 's@+@ @g;s@%@\\x@g' file | xargs -0 printf "%b"

Source

sed with echo -e

$ sed -e's/%\([0-9A-F][0-9A-F]\)/\\\\\x\1/g' file | xargs echo -e

Source

sed with alias

For convenience, you may want to use an alias:

$ alias urldecode='sed "s@+@ @g;s@%@\\\\x@g" | xargs -0 printf "%b"'

If you want to decode, you can now simply use:

$ echo "http%3A%2F%2Fwww" | urldecode
http://www

Source

Bash

input="http%3A%2F%2Fwww"
decoded=$(printf '%b' "${input//%/\\x}")

Source

To handle pluses (+) correctly, replace them with spaces using sed:

decoded=$(input=${input//+/ }; printf "${input//%/\\x}")

Bash + urlencode() + urldecode() Functions

urlencode() {
    # urlencode <string>
    local length="${#1}"
    for (( i = 0; i < length; i++ )); do
        local c="${1:i:1}"
        case $c in
            [a-zA-Z0-9.~_-]) printf "$c" ;;
            *) printf '%%%02X' "'$c" ;;
        esac
    done
}

urldecode() {
    # urldecode <string>

    local url_encoded="${1//+/ }"
    printf '%b' "${url_encoded//%/\\x}"
}

Sources:

bash + xxd

urlencode() {
  local length="${#1}"
  for (( i = 0; i < length; i++ )); do
    local c="${1:i:1}"
    case $c in
      [a-zA-Z0-9.~_-]) printf "$c" ;;
    *) printf "$c" | xxd -p -c1 | while read x;do printf "%%%s" "$x";done
  esac
done
}

Sources:

PHP

$ echo oil+and+gas | php -r 'echo urldecode(fgets(STDIN));' // Or: php://stdin
oil and gas

Source

PHP Library

php -r 'echo urldecode("oil+and+gas");'

Source

Perl

decoded_url=$(perl -MURI::Escape -e 'print uri_unescape($ARGV[0])' "$encoded_url")

Source

Perl to Process File

perl -i -MURI::Escape -e 'print uri_unescape($ARGV[0])' file

Source

awk

awk -niord '{printf RT?$0chr("0x"substr(RT,2)):$0}' RS=%..

Sources:

Python 2 urllib.unquote

The urllib.unquote is a special function in Python’s built-in standard library urllib that does what you need:

decoded_url=$(python2 -c 'import sys, urllib; print urllib.unquote(sys.argv[1])' "$encoded_url")

You can also use it to modify a file:

python2 -c 'import sys, urllib; print urllib.unquote(sys.stdin.read())' <file >file.new &&
mv -f file.new file

Source: https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding

Python 3 urllib.parse.unquote

If you run Python 3 on your system (like most people would), use the alternative function urllib.parse.unquote. To check your version, visit this article.

decoded_url=$(python3 -c 'import sys, urllib.parse; print(urllib.parse.unquote(sys.argv[1]))' "$encoded_url")

Again, you can use the function to process a file as follows:

python3 -c 'import sys, urllib; print(urllib.parse.unquote(sys.stdin.read()))' <file >file.new &&
mv -f file.new file

Source: https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding

Perl URI::Escape

The URI::Escape solves the problem of URL decoding for Perl users.

decoded_url=$(perl -MURI::Escape -e 'print uri_unescape($ARGV[0])' "$encoded_url")

You can use the function to process a file as follows:

perl -i -MURI::Escape -e 'print uri_unescape($ARGV[0])' file

Source: https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding

Perl One-Liner Without Installing Modules

$ perl -pe 's/\%(\w\w)/chr hex $1/ge'

Here’s a usage example:

$ echo '%21%22' |  perl -pe 's/\%(\w\w)/chr hex $1/ge'
!"

Source: https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding

Bash Regex

$ function urldecode() { : "${*//+/ }"; echo -e "${_//%/\\x}"; }

Now, you can use the function as a command like this:

$ urldecode https%3A%2F%2Fgoogle.com%2Fsearch%3Fq%3Durldecode%2Bbash
https://google.com/search?q=urldecode+bash

If you need to assign some variables, use this strategy:

$ x="http%3A%2F%2Fstackoverflow.com%2Fsearch%3Fq%3Durldecode%2Bbash"
$ y=$(urldecode "$x")
$ echo "$y"
http://stackoverflow.com/search?q=urldecode+bash

Source: https://stackoverflow.com/questions/6250698/how-to-decode-url-encoded-string-in-shell

GNU Awk

#!/usr/bin/awk -fn
@include "ord"
BEGIN {
   RS = "%.."
}
{
   printf "%s", $0
   if (RT != "") {
      printf "%s", chr("0x" substr(RT, 2)) 
   }
}

Source: https://stackoverflow.com/questions/6250698/how-to-decode-url-encoded-string-in-shell

References