Learn CyberChef: A Beginner’s Guide to Encoding/Decoding

Okay, here’s a comprehensive article on CyberChef, focusing on its use for encoding and decoding, aimed at beginners but covering a wide range of its capabilities:

Learn CyberChef: A Beginner’s Guide to Encoding/Decoding (and Much More!)

CyberChef is a powerful, versatile, and completely free web application often described as the “Cyber Swiss Army Knife.” Developed by the UK’s Government Communications Headquarters (GCHQ), it provides a vast array of tools for encoding, decoding, encrypting, decrypting, compressing, decompressing, hashing, and analyzing data. It’s a vital resource for cybersecurity professionals, software developers, data analysts, and anyone who needs to manipulate data in various formats. This guide will walk you through the fundamentals of CyberChef, focusing on its encoding and decoding capabilities, but also touching on other powerful features.

Why CyberChef?

Before we dive into the specifics, let’s understand why CyberChef stands out:

• No Installation Required: It’s a web application. You access it directly through your browser (https://gchq.github.io/CyberChef/). No downloads, no installations, no dependencies. This makes it incredibly portable and accessible.
• Client-Side Processing: All operations happen within your browser. Your data never leaves your computer unless you explicitly choose to save it. This is crucial for privacy and security, especially when dealing with sensitive information.
• Intuitive Interface: CyberChef uses a drag-and-drop interface with a clear “recipe” system. You build a sequence of operations (the recipe) and apply it to your input data. This makes it easy to experiment and see the results in real-time.
• Huge Library of Operations: CyberChef boasts hundreds of operations, covering everything from simple base64 encoding to complex cryptographic algorithms.
• Open Source and Extensible: The source code is available on GitHub, allowing anyone to contribute, audit, or modify the tool. You can even create your own custom operations if needed.
• Regularly Updated: GCHQ actively maintains and updates CyberChef, adding new features and operations frequently.
• Offline Capability: While it’s a web app, CyberChef can be used offline. You can download the HTML file and open it in your browser even without an internet connection. This is incredibly useful for fieldwork or situations with limited connectivity.

Getting Started: The CyberChef Interface

When you open CyberChef, you’ll see a clean and intuitive interface divided into several key sections:

1. Input: This is where you paste or type the data you want to process. You can also drag and drop files directly into this area. CyberChef automatically tries to detect the input type (text, hex, file, etc.).

2. Operations: This is the heart of CyberChef. It’s a searchable list of all available operations, categorized for easy browsing. You can use the search bar to quickly find the operation you need (e.g., “base64,” “AES,” “XOR”).

3. Recipe: This is where you build your data manipulation workflow. You drag operations from the “Operations” section into the “Recipe” section to create a sequence of steps. Each step is applied in order, and you can easily reorder, modify, or delete steps.

4. Output: This area displays the result of applying your recipe to the input data. The output updates in real-time as you modify the recipe.

5. Bake/Auto Bake: The “Bake” button manually executes the recipe. The “Auto Bake” toggle (highly recommended) automatically re-executes the recipe whenever you make changes to the input, recipe, or operation parameters. This provides immediate feedback and makes experimentation much easier.

6. Save/Load Recipe: You can save your recipes as JSON files and load them later, allowing you to reuse complex workflows. This is particularly useful for frequently used operations or for sharing recipes with others.

7. Input/Output Controls: These controls allow you to:

   • Load from File: Load data directly from a file on your computer.
   • Save to File: Save the output to a file.
   • Copy to Clipboard: Copy the input or output to your clipboard.
   • Clear Input/Output: Clear the contents of the input or output areas.
   • Swap Input/Output: Quickly swap the contents of the input and output areas. This is useful for reversing a process.
   • Fork: Create a new branch in your recipe, allowing you to experiment with different operations on the same input data.
   • Magic: CyberChef’s “Magic” operation attempts to automatically detect the encoding/encryption and suggest a recipe to decode/decrypt the input. This is a great starting point for unknown data.

Fundamental Encoding and Decoding Operations

Let’s start with the core of CyberChef: encoding and decoding. These operations transform data from one representation to another.

1. Base64

• What it is: Base64 is a very common encoding scheme that represents binary data in an ASCII string format. It’s frequently used to transmit data over channels that only reliably support text, such as email or URLs. Base64 uses a set of 64 characters (A-Z, a-z, 0-9, +, /) and padding (=) to represent data.
• CyberChef Operations:
  • To Base64: Encodes data to Base64. You can specify the character set (standard, URL-safe, etc.).
  • From Base64: Decodes Base64-encoded data. CyberChef automatically handles padding and different character sets.
• Example:
  • Input: Hello, World!
  • To Base64 Recipe: To Base64()
  • Output: SGVsbG8sIFdvcmxkIQ==
  • Input: SGVsbG8sIFdvcmxkIQ==
  • From Base64 Recipe: From Base64()
  • Output: Hello, World!

2. Hexadecimal (Hex)

• What it is: Hexadecimal is a base-16 numbering system that uses 16 symbols (0-9 and A-F) to represent values. It’s commonly used in computer science to represent binary data in a more compact and human-readable format. Each hex digit represents four bits.
• CyberChef Operations:
  • To Hex: Converts data to a hexadecimal string. You can specify options like delimiters (spaces, commas, etc.), and whether to include the “0x” prefix.
  • From Hex: Converts a hexadecimal string back to its original data.
• Example:
  • Input: CyberChef
  • To Hex Recipe: To Hex()
  • Output: 437962657243686566
  • Input: 437962657243686566
  • From Hex Recipe: From Hex()
  • Output: CyberChef

3. URL Encoding (Percent Encoding)

• What it is: URL encoding replaces unsafe ASCII characters in a URL with a percent sign (%) followed by two hexadecimal digits. This ensures that URLs are transmitted correctly, as certain characters have special meanings in URLs (e.g., spaces, ampersands, question marks).
• CyberChef Operations:
  • URL Encode: Encodes a string for use in a URL.
  • URL Decode: Decodes a URL-encoded string.
• Example:
  • Input: https://example.com/search?q=cyber chef
  • URL Encode Recipe: URL Encode()
  • Output: https%3A%2F%2Fexample.com%2Fsearch%3Fq%3Dcyber%20chef
  • Input: https%3A%2F%2Fexample.com%2Fsearch%3Fq%3Dcyber%20chef
  • URL Decode Recipe: URL Decode()
  • Output: https://example.com/search?q=cyber chef

4. HTML Entities

• What it is: HTML entities are used to represent characters that have special meanings in HTML (e.g., <, >, &) or characters that are not easily typed on a keyboard. They start with an ampersand (&) and end with a semicolon (;).
• CyberChef Operations:
  • Encode HTML Entities: Converts characters to their HTML entity equivalents.
  • Decode HTML Entities: Converts HTML entities back to their corresponding characters.
• Example:
  • Input: <p>Hello & World</p>
  • Encode HTML Entities Recipe: Encode HTML Entities()
  • Output: <p>Hello & World</p>
  • Input: <p>Hello & World</p>
  • Decode HTML Entities Recipe: Decode HTML Entities()
  • Output: <p>Hello & World</p>

5. Character Encoding (UTF-8, UTF-16, ASCII, etc.)

• What it is: Character encoding defines how characters are represented as numerical values (bytes) in a computer. Different encodings use different mappings. UTF-8 is the most common encoding on the web.
• CyberChef Operations:
  • Decode Text: This operation attempts to decode the input based on a specified character encoding.
  • Encode Text: This operation encodes text using a specified character encoding.
• Example:
  • If you have a file that’s garbled because it’s being interpreted with the wrong encoding, you can use Decode Text and try different encodings (UTF-8, UTF-16, Latin-1, etc.) until you find the correct one.

6. Morse Code

• What it is: A method of transmitting text information as a series of on-off tones, lights, or clicks.
• CyberChef Operations:
  • To Morse Code: Converts plain text to morse code.
  • From Morse Code: Converts morse code to plain text.
• Example:
  • Input: SOS
  • To Morse Code Recipe: To Morse Code()
  • Output: ... --- ...
  • Input: ... --- ...
  • From Morse Code Recipe: From Morse Code()
  • Output: SOS

7. Binary, Octal, Decimal

• What it is: Different number systems. Binary (base-2) uses only 0 and 1. Octal (base-8) uses 0-7. Decimal (base-10) is the standard number system we use every day.
• CyberChef Operations:
  • To Binary: Converts data to a binary string.
  • From Binary: Converts a binary string to data.
  • To Octal: Converts data to an octal string.
  • From Octal: Converts an octal string to data.
  • To Decimal: Converts data to a decimal string.
  • From Decimal: Converts a decimal string to data.

8. Regular Expressions (Regex)
* What it is: Regular expressions are powerful pattern-matching tools. They allow you to search for, extract, and replace text based on complex patterns. CyberChef has excellent regex support.
* CyberChef Operations:
* Regular expression: This operation allows you to perform various regex operations:
* Find / List Matches: Find all occurrences of a pattern.
* Extract: Extract specific parts of the matched text (using capture groups).
* Replace: Replace matched text with a different string.
* Split: Split the input string based on a regex delimiter.
* Flags: You can set various regex flags, such as:
* g (global): Find all matches, not just the first one.
* i (case-insensitive): Ignore case when matching.
* m (multiline): Treat the input as multiple lines.
* Example: Extracting email addresses from text:
* Input: Contact us at [email protected] or [email protected].
* Recipe: Regular expression('User defined', '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}', true, true, false, false, false, 'List matches')
* Output:
[email protected]
[email protected]

Working with XOR (Exclusive OR)

XOR is a bitwise operation that’s fundamental to many cryptographic algorithms. It’s a simple but powerful tool for encoding and decoding.

• What it is: XOR takes two bits as input and outputs 1 if the bits are different, and 0 if they are the same.
  • 0 XOR 0 = 0
  • 0 XOR 1 = 1
  • 1 XOR 0 = 1
  • 1 XOR 1 = 0

• CyberChef Operations:

  • XOR: Performs a bitwise XOR operation. You can specify the key in various formats (text, hex, base64, etc.) and choose different key lengths (single byte, multi-byte, repeating key).
  • XOR Brute Force: Attempts to XOR the input with all possible keys of a specified length. This is incredibly useful for breaking simple XOR ciphers.

• Example (Simple XOR Encryption):

  • Input: Secret Message
  • Key: Key
  • Recipe: XOR({'option':'UTF8','string':'Key'},'Standard',false)

  • Output: (Encoded data – will appear as gibberish)

  • Input: (Encoded data from above)

  • Key: Key
  • Recipe: XOR({'option':'UTF8','string':'Key'},'Standard',false)
  • Output: Secret Message
    (Notice that the same operation with the same key both encodes and decodes the data. This is a key property of XOR.)

• XOR Brute Force Example:

  • Input: (Some data you suspect is XOR-encoded with a single-byte key)
  • Recipe: XOR Brute Force(1, 1, '80', '00', 'None', false, false, false, 'Alphanumeric')
  • Explanation:
    • 1, 1: Brute-force keys from length 1 to length 1 (single-byte).
    • '80', '00': This is the range of characters to try (hexadecimal 80 to 00, covering all 256 possible byte values). Often, you can reduce this range if you know something about the key (e.g., only printable ASCII characters).
    • 'None': No delimiter between key bytes (since we’re using single-byte keys).
    • false, false, false: Various options, not relevant for this simple case.
    • 'Alphanumeric': This is a crib. CyberChef will highlight results that contain a high proportion of alphanumeric characters, making it easier to identify the correct key.

  CyberChef will output a long list of possible decryptions, one for each possible key. You’ll need to scan through the output, looking for something that makes sense (using the crib highlighting to help). This is a crucial technique in cryptanalysis.

Basic Hashing

Hashing algorithms generate a fixed-size “fingerprint” (the hash) of a piece of data. They are one-way functions: it’s computationally infeasible to reverse the process and recover the original data from the hash. Hashes are used for data integrity verification, password storage, and other security applications.

• CyberChef Operations (Examples):

  • MD5: Generates an MD5 hash. (Note: MD5 is considered cryptographically broken and should not be used for security-sensitive applications.)
  • SHA1: Generates an SHA-1 hash. (Also considered weak and should be avoided for new applications.)
  • SHA256: Generates an SHA-256 hash (a strong and widely used hashing algorithm).
  • SHA512: Generates an SHA-512 hash (even stronger than SHA-256).
  • HMAC: Calculates a Hash-based Message Authentication Code (HMAC). HMACs use a secret key in addition to the data, providing both integrity and authenticity.

• Example (SHA-256):

  • Input: This is a test.
  • Recipe: SHA256()
  • Output: f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2

Beyond Encoding/Decoding: Other Powerful Features

While encoding and decoding are core functionalities, CyberChef offers much more. Here’s a glimpse of some other valuable operations:

• Cryptography:

  • AES Encrypt: Encrypt data using the Advanced Encryption Standard (AES) algorithm.
  • AES Decrypt: Decrypt AES-encrypted data.
  • RSA Encrypt: Encrypt data using the RSA algorithm (public-key cryptography).
  • RSA Decrypt: Decrypt RSA-encrypted data.
  • Generate PGP Key Pair: Generate a keypair to be used with Pretty Good Privacy.
  • …and many more (Blowfish, DES, RC4, etc.).

• Data Formats:

  • Parse JSON: Parse a JSON string into a structured object.
  • Parse XML: Parse an XML string into a structured object.
  • Parse CSV: Parse Comma-Separated Values (CSV) data.
  • Parse JWT: Decode and verify JSON Web Tokens (JWTs).

• Networking:

  • Parse IP Address: Extract information from an IP address (IPv4 or IPv6).
  • Parse User Agent: Extract information from a User-Agent string (browser, operating system, etc.).
  • Parse Domain: Extract information from a domain name (e.g., TLD, registered domain).

• Compression:

  • Gzip Compress
  • Gzip Decompress
  • Zip
  • Unzip

• Utilities:

  • Count occurrences: Find the number of times a particular string appears in input data.
  • Remove whitespace
  • Reverse: Reverse the input string.
  • Sort: Sort lines of text alphabetically or numerically.
  • Unique: Remove duplicate lines.
  • Entropy: Calculate the entropy of the input data (a measure of randomness).

• Image Operations

  • Extract EXIF
  • Render Image
  • Rotate Image

“Magic” Operation: Your First Line of Defense

When confronted with unknown data, the “Magic” operation is your best friend. It analyzes the input and tries to guess the encoding or encryption used. It then suggests a recipe to decode or decrypt the data. It’s not always perfect, but it’s incredibly effective as a starting point.

• How to Use:
  • Paste your unknown data into the Input area.
  • Drag the “Magic” operation (under the “Flow Control” category) into the Recipe area.
  • Set the “Intensity” and “Extensive” parameters. Higher intensity and extensive search will take longer but may be more accurate.
• Example
  • Input: gAN9cQA= (unknown encoding)
  • Magic Recipe: Magic(2,false,'Normal')
  • Output: Cyberchef will suggest a recipe, which, in this case, is From Base64('A-Za-z0-9+/=',true) followed by From Base85('<~,~>'). Cyberchef has correctly determined that the input is double encoded.

Building Complex Recipes

The real power of CyberChef comes from combining multiple operations into a recipe. You can chain together encoding, decoding, filtering, and other operations to create complex data manipulation workflows.

Example: Decoding a Double-Encoded String

Let’s say you encounter a string that you suspect is first encoded in Base64 and then URL-encoded.

• Input: SGVsbG8lMjBXb3JsZCUyMQ%3D%3D
• Recipe:
  1. URL Decode()
  2. From Base64()
• Output: Hello World!

Example: Extracting Data from a Log File

Imagine a log file with lines like this:

2023-10-27 10:15:30 [INFO] User logged in: user123
2023-10-27 10:16:00 [ERROR] Invalid password for user: admin
2023-10-27 10:17:15 [INFO] User logged out: user123

You want to extract all usernames after “User logged in:”.

• Input: (The log file content)

• Recipe:

  1. Regular expression('User defined', 'User logged in: (.*)', true, true, false, false, false, 'List matches') (This extracts the username using a capture group)

• Output:
  user123

Tips and Best Practices

• Start Simple: Begin with basic encoding/decoding operations to get familiar with the interface.
• Use Auto Bake: Enable “Auto Bake” for real-time feedback.
• Experiment: Don’t be afraid to try different operations and see what happens.
• Use “Magic”: The “Magic” operation is a great starting point for unknown data.
• Break Down Complex Tasks: Decompose complex problems into smaller, manageable steps.
• Save Useful Recipes: Save frequently used recipes for later use.
• Read the Documentation: CyberChef has extensive online documentation with detailed explanations of each operation.
• Understand the Underlying Concepts: A good understanding of encoding schemes, cryptography, and regular expressions will greatly enhance your ability to use CyberChef effectively.
• Use Forks Forking allows you to create separate branches in your recipe. This lets you try different operations without affecting the main flow, great for comparing outputs.
• Understand Input/Output Formats Know if you should be working with strings, hex, base64, etc. and use appropriate conversions when needed.

Conclusion

CyberChef is an incredibly powerful and versatile tool that’s essential for anyone working with data in various formats. This guide has provided a comprehensive introduction to its encoding and decoding capabilities, as well as a glimpse into its other powerful features. By mastering CyberChef, you’ll gain a valuable skill set that will be useful in a wide range of tasks, from cybersecurity analysis to software development to data manipulation. The best way to learn is by doing, so open up CyberChef and start experimenting!