tom-lord.github.io

10 More New Features in Ruby v2.5

2017-12-11T00:00:00+00:00

With the upcoming release of ruby v2.5 scheduled (as per tradition) for 25th December, it’s good to know what’s changed in the language - so you can take advantage any new (or refined) features.

This popular blog post by Junichi Ito highlighted 10 new features; but since there are so many improvements to the language, let’s dive in and unravel 10 more handpicked highlights!

1. More public `Module` methods

(Features #14132 and #13133).

Module#attr, attr_accessor, attr_reader, attr_writer, define_method, alias_method, undef_method and remove_method are now all public.

For example:

# Ruby v2.4
Integer.alias_method :plus, :+
#=> NoMethodError: private method `alias_method' called for Integer:Class

# Workaround 1:
Integer.send(:alias_method, :plus, :+)

# Workaround 2:
class Integer
  alias_method :plus, :+
end

# Ruby v2.5
Integer.alias_method :plus, :+

This update follows nicely from similar changes back in Ruby v2.1.0, where Module#include and Module#prepend were made public.

I think this is a great example of ruby being a community-driven language: Its design has evolved over time, influenced by how its users want it to be; not set in stone due to the original designers’ opinions.

2. `String#start_with?` supports regexp

(Feature #13712).

This enhancement provides some nice syntactic sugar, which could help prevent a common (and sometimes security-related) mistake in ruby:

# Checking whether a string starts with a lower-case letter

string.match?(/^[a-z]/) # BAD!
string.match?(/\A[a-z]/) # Good

The mistake arrises because, unlike most other languages, the ^ anchor means “starts of line” in ruby; not “start of string”. (That’s what \A is for.)

So, that first line of code would also return true when string = "123\ntest".

Using the new String#start_with? method, such a mistake wouldn’t happen:

# Ruby v2.5
string.start_with?(/[a-z]/) # Also good

3. Improvements to `binding.irb`

(Features #13099 and #14124).

For a few years now, the recommended ruby toolkit for developer consoles and debuggers has been dominated by pry, byebug, or even some hybrid of the above.

…Which is a little odd, since ruby already comes with a pretty good built-in REPL: irb (“Interactive Ruby”).

But as of ruby v2.5, we see two enhancements to the library that help bridge the gap, and make it a little less likely to feel the need for pry in every application:

require 'irb' is no longer needed in your code, in order to invoke binding.irb.
Show source around binding.irb is now shown on startup.

# Ruby v2.4

# test.rb:
require 'irb'
def test
  binding.irb
end
test

# Running the file yields:
irb(main):001:0>

# Ruby v2.5:
# test.rb:
def test
  binding.irb
end
test

# Running the file yields:
From: test.rb @ line 2 :

    1: def test
 => 2:   binding.irb
    3: end
    4: test

irb(main):001:0>

4. `Integer.sqrt` added

(Feature #13219)

Quite often in algorithms, you need to find “the largest integer less than sqrt(n)”. For example, a simple algorithm to check whether n is prime is to simply try dividing n by all primes up to sqrt(n).

This operation can now be executed more succinctly, within the core library methods:

# Ruby v2.4
Math.sqrt(4) #=> 2.0
Math.sqrt(4).to_i #=> 2

Math.sqrt(10) #=> 3.1622776601683795
Math.sqrt(10).to_i #=> 3

# Ruby v2.5
Integer.sqrt(4) #=> 2
Integer.sqrt(10) #=> 3

5. `String#casecmp` and `casecmp?` now return nil for non-string arguments

(…instead of raising a TypeError)

(Bug #13312).

Fairly self-explanatory:

# Ruby v2.4
"foo".casecmp 123
#=> TypeError: no implicit conversion of Integer into String

# Ruby v2.5
"foo".casecmp 123
#=> nil

Changes like this usually slip under the radar, as they are not “exciting” features. But it just goes to show: There’s still all sorts of simple room for improvement in the language; and sometimes the change you wish to make may not even be very complicated!

But all of these many incremental changes add up, and are what makes modern ruby such a robust, stable language.

6. `mathn.rb` removed from stdlib

(Feature #10169).

It may sound odd to announce the “removal” of a library as a new feature, but please hear me out!…

mathn is an unusual library. It not only adds to, but also changes, the behaviour of Integers!

When you require 'mathn', this in turn loads other libraries such as require 'prime', which adds the methods: Integer.each_prime, Integer.from_prime_division, Integer#prime? and Integer#prime_division.

However, more bizarrely, it also redefines Integer#/ to be an alias for Numeric#quo!

1/3 # => 0
(1/3).class #=> Integer

# In Ruby v2.5, this line will `raise LoadError: cannot load such file -- mathn`
# (Unless you also install the deprecated `mathn` gem.)
require 'mathn'
1/3 # => (1/3)
(1/3).class #=> Rational

This isn’t always a problem, but it certainly doesn’t conform the the principle of least surprise.

For example, there was a case about 9 years ago where the Rubinius VM essentially “blew up” due to some obscure code simply running require 'mathn'. Even today, people are sometimes left confused by different behaviour on development vs production servers, due to some production-specific dependency loading the library!

This behaviour-changing aspect of the mathn library has been a known issue for several years, now. The library was officially “deprecated” as of Ruby v2.2, but that doesn’t stop people from using it of course!

However, thanks to this change in Ruby v2.5, you can now only load mathn as a gem. This finally means:

Developers will need to bundle a deprecated gem into their project, in order to use it. (Which means usage is much less likely!)
…And therefore, even if some dependency in your project does require mathn, you will have traceability of the root culprit in the Gemfile.lock!

7. Default template file encoding is changed from ASCII-8BIT to UTF-8 in erb command

(Bug #14095).

Similar to #5 above, this just goes to show how well the open source community works to improve the language. What started out as a discussion on Reddit was then raised with the core developers, and promptly fixed. And, once again, the code fix was simple.

This (obscure) bug was in fact a lingering legacy of Ruby v1.9’s implementation! However, its fix could prove to be quite useful in conjunction with other changes in Ruby v2.5.

8. `Hash#slice` added

(Feature #8499).

The ActiveSupport library, which comes bundled with the popular Ruby on Rails framework (but can also be used in isolation) provides many - often controversial - extensions to Ruby’s core classes.

Over the past few years, however, Ruby has gradually been cherry-picking the “best” features and merging them into the core language. Hash#slice is one such method; continuing the trend.

# Ruby v2.5
{a: 1, b: 2, c: 3}.slice(:a, :b)
#=> {:a=>1, :b=>2}

9. `SecureRandom.alphanumeric` added

(Feature #10849).

One thing that’s always struck me as a little odd in ruby, given its vastly rich core library methods, is its lack of a clear “random string” generator.

This StackOverflow question, for example, demonstrates just some of the many (often surprising!) popular methods used by developers, to generate a “random string”. Here are a few of them:

# 1.
(0...50).map { ('a'..'z').to_a[rand(26)] }.join

# 2.
('a'..'z').to_a.shuffle[0,8].join

# 3.
rand(36**length).to_s(36)

# 4.
require 'securerandom'
SecureRandom.hex

Each approach has various pros and cons with regard to its flexibility, readability and entropy (“true randomness”). For instance, only option 4 above should be considered cryptographically secure. Other SecureRandom methods include:

SecureRandom.base64 # e.g. "/qWJPsvoxnSe17HrTlzQ7Q=="
SecureRandom.urlsafe_base64 # e.g. "IINbXW4YKjfoncJMpP-CkQ"
SecureRandom.random_bytes # e.g. "\t\x85\xB9\x9C\xD1\f\xD3\xE6t\xB4S^\v-\xFFo"
SecureRandom.random_number # e.g. 0.8993676724126768

Due to the common use case of needing a purely alphanumeric string, the new SecureRandom.alphanumeric method fits in nicely with this collection.

Unfortunately though, if you need to generate a cryptographically secure random string with some other specific characters, you’ll need still to write something a little clunky - e.g.

# 20 random chars; chosen from a, b, c, d, e, v, w, x, y and z
(1..20).flat_map { [*'a'..'e', *'v'..'z'].sample(1, random: SecureRandom) }

… Or if you’re just looking for a clean syntax for arbitrary random strings, why not check out my regexp-examples library?

10. `Net::HTTP::STATUS_CODES`

(Feature #12935).

Ruby v2.5 has now defined a hash of all HTTP status codes!

Previously, this was only recorded in documentation, and it was left up to web frameworks such as Rails to define their own “human-friendly” mappings.

Additionally, definitions of the following status codes have been added to ruby:

# Ruby v2.5
require 'net/http/status_codes'
Net::HTTP::STATUS_CODES
{
  # ...
  208 => 'Already Reported',
  308 => 'Permanent Redirect',
  421 => 'Misdirected Request',
  506 => 'Variant Also Negotiates',
  508 => 'Loop Detected',
  510 => 'Not Extended',
  # ...
}

Bonus: Unicode 10.0 support

(Feature #13685).

Keeping up with the new unicode standard, released in June 2017, Ruby v2.5 has been updated to support the new characters.

The new characters include a Bitcoin symbol and 56 new emojis!

Conclusion

The ruby language has increasingly stabilised over the past few years. And while it’s fair to say that these changes are far less revolutionary than what we saw back in Ruby v2.0 for instance, there is still a lot going on within this active community.

Ruby v2.5 will be released on Christmas Day, but you can download a pre-release version today and try it out!

Missy Elliott’s Reciprocal Cipher

2015-03-29T00:00:00+00:00

MissyElliott.encode("Example") # => "\xAE\xF0\xBC\xA4\xF8\xE4\xAC"

#"Example"
#--> ["E", "x", "a", "m", "p", "l", "e"]
#--> [69, 120, 97, 109, 112, 108, 101]
#--> ["01000101", "01111000", "01100001", "01101101", "01110000", "01101100", "01100101"]
# Shift yo bits down
#--> ["10001010", "11110000", "11000010", "11011010", "11100000", "11011000", "11001010"]
# Flip it
#--> ["01110101", "00001111", "00111101", "00100101", "00011111", "00100111", "00110101"]
# And reverse it
#--> ["10101110", "11110000", "10111100", "10100100", "11111000", "11100100", "10101100"]
#--> [174, 240, 188, 164, 248, 228, 172]
#--> ["\xAE", "\xF0", "\xBC", "\xA4", "\xF8", "\xE4", "\xAC"]
#--> "\xAE\xF0\xBC\xA4\xF8\xE4\xAC"

This is an excerpt from the README of a silly ruby gem I published, aptly named: missy_elliott. It is obvious, at a glance, that this encoding algorithm is easily reversible: You simply repeat the same 3 steps (“shift, flip, reverse”) in reverse order (“reverse, flip, un-shift”).

However, after further investigation, I noticed something strange: MissyElliott.encode, and MissyElliott.decode - despite having different implementations - are actually doing exactly the same thing!!

MissyElliott.encode("How is this even possible?!")
# => "\xF6\x84\x88\xFD\xB4\x98\xFD\xE8\xF4\xB4\x98\xFD\xAC\xC8\xAC\xC4\xFD\xF8\x84\x98\x98\xB4\xDC\xE4\xAC\x81\xBD"

MissyElliott.encode("\xF6\x84\x88\xFD\xB4\x98\xFD\xE8\xF4\xB4\x98\xFD\xAC\xC8\xAC\xC4\xFD\xF8\x84\x98\x98\xB4\xDC\xE4\xAC\x81\xBD")
# => "How is this even possible?!"

MissyElliott.decode("How is this even possible?!")
# => "\xF6\x84\x88\xFD\xB4\x98\xFD\xE8\xF4\xB4\x98\xFD\xAC\xC8\xAC\xC4\xFD\xF8\x84\x98\x98\xB4\xDC\xE4\xAC\x81\xBD"

MissyElliott.decode("\xF6\x84\x88\xFD\xB4\x98\xFD\xE8\xF4\xB4\x98\xFD\xAC\xC8\xAC\xC4\xFD\xF8\x84\x98\x98\xB4\xDC\xE4\xAC\x81\xBD")
# => "How is this even possible?!"

This completely threw me off when I first saw it - I figured there must be a bug in my code. But it’s true!

If you’d like to have a go at proving this for yourself, stop reading now. I’ll show my answer after the following message from my sponsor:

Reciprocal Ciphers

A reciprocal cipher is an encoding algorithm that is the inverse of itself. Perhaps the simplest, best-known example of such a cipher is ROT-13; a special case of the Caesar cipher.

The reason why ROT-13 is reciprocal is quite obvious: We shift each letter of the input down (or up!) 13 places in the alphabet. And, since there are 26 letters in the alphabet, repeating the process gets us back to where we started. For example:

"flip me" --> "syvc zr" --> "flip me"

But how on Earth does Missy Elliott manage this, with her much more complicated encoding?! Here’s one example, to show the reciprocal encoding in action:

ORIGINAL: 10011101

shift: 00111011

flip: 11000100

reverse: 00100011 (encoded)

shift: 01000110

flip: 10111001

reverse: 10011101 (twice encoded == ORIGINAL!!)

Is this always true? Can we prove it? Yes - here’s what I came up with:

We only need to consider what happens to an individual bit, when applying the Missy Elliot algorithm (twice). Each bit an be precisely described by two things: its value (1 or 0), and its position (how many bits are to the left/right).

Without loss of generality, let’s consider what happens to a single bit, of value B (the opposite of which is b), which has x bits to its left (and for the sake of clarity, y bits to its right). The following syntax should be fairly self explanatory:

*There is a slight edge case here, which I have omitted: What happens when x=0, i.e. the bit is “shifted” onto the back of the list? However, this is a fairly trivial edge case to cover; I leave this as an exercise for the reader. (I’ve always wanted to use that phrase, after having it drilled into me at university!)

Missy Elliott’s Graph

Missy Elliott’s algorithm essentially works by mapping each character’s ASCII code to an “encoded” number, then converting it back to the corresponding character in the ASCII table. Since the algorithm is reciprocal, this creates a 1-to-1 pairing between ASCII codes when repeating the encoding.

For example:

= 00000000 <--> 11111111 = 255
= 00000001 <--> 10111111 = 191
= 00000010 <--> 11011111 = 223

Do you wonder what it would look like to plot all 256 points on a scatter graph? Well, I did:

The obvious property that this graph shows is: If x<128, then Encoding(x) >=128 (and vice versa). Proving this is quite easy:

Using similar technique to the above proof, only in this case B represents 7 digits, rather than just 1 (and b represents its “flipped” value), consider what happens when we apply the Missy Elliott encoding to any number < 128:

In other words, if the original value starts with a 0 (i.e. is < 128) then its encoded value must start with a 1 (i.e. is >= 128). However, there is a far more interesting (less obvious) feature of this graph: it always oscillates in value…

Encoding(0) = 255 > Encoding(1) = 191 < Encoding(2) = 223 > Encoding(3) = 159 < ...

… Except for one point, in the middle:

Encoding(126) = 192 > Encoding(127) = 128 > Encoding(128) = 127 > Encoding(129) = 64
                                         ^^^

This can be visualised by taking a subsection of the above graph:

In fact, this sequence has an even crazier property hiding beneath the surface. Let’s only look at the second half of the mappings, i.e.

Encoding(128), Encoding(129), ..., Encoding(255)

This sub-sequence is equal to:

127, 63, 95, 31, 111, 47, 79, 15, 119, 55, 87, 23, 103, 39, 71, 7, 123, 59, 91, 27, 107, 43, 75, 11, 115, 51, 83, 19, 99, 35, 67, 3, 125, 61, 93, 29, 109, 45, 77, 13, 117, 53, 85, 21, 101, 37, 69, 5, 121, 57, 89, 25, 105, 41, 73, 9, 113, 49, 81, 17, 97, 33, 65, 1, 126, 62, 94, 30, 110, 46, 78, 14, 118, 54, 86, 22, 102, 38, 70, 6, 122, 58, 90, 26, 106, 42, 74, 10, 114, 50, 82, 18, 98, 34, 66, 2, 124, 60, 92, 28, 108, 44, 76, 12, 116, 52, 84, 20, 100, 36, 68, 4, 120, 56, 88, 24, 104, 40, 72, 8, 112, 48, 80, 16, 96, 32, 64, 0

Perfect Oscillating Sequences

Disclaimer: I have no idea if sequences with this property have ever been named before; I certainly cannot find one! I invented the name “perfect oscillating”, but if you feel an alternative name is more suiting, or are aware of pre-existing name, please let me know in the comments!

Let’s take a look at a few sub-sequences of the above:

(a_2n) = 63, 31, 47, 15, 55, 23, 39, 7, 59, 27, ...
(a_3n) = 95, 47, 119, 23, 71, 59, 107, 11, 83, ...
(a_4n) = 31, 15, 23, 7, 27, 11, 19, 3, 29, 13, ...
(a_2n-1) = 127, 95, 111, 79, 119, 87, 103, 71, 123, ...
(a_5n-3) = 63, 79, 23, 123, 43, 83, 3, 109, 53, 69, ...

Any sub-sequence of the form (a_xn+y) also oscillates!!

As it turns out, Missy Elliott’s song is all about a novel way to generate the sequence: A030109[7]. Who would have guessed?!

In fact, there’s one very useful application for sequences like these - the answer might come as a surprise!

To keep things simple from here on, let’s use a somewhat shorter sequence. The following perfect oscillating sequence can also be generated using the Missy Elliott algorithm (on the numbers 0 - 7), and adding 1 to each term:

8, 4, 6, 2, 7, 3, 5, 1

I found a clue for how this sequence is used, in the comments for A049773. To summarise, this is the “optimally fair” starting line-up for ranked players in a knock-out tournament! Assuming the favourite always wins, the result of such a tournament would look like this:

Missy Elliott truly is a lyrical genius, after all.

Phew, well that got a little side-tracked from the original purpose of this blog post!

Was it worth it?

Reverse Engineering Regular Expressions

2015-03-19T00:00:00+00:00

I recently published a powerful ruby gem on Github: regexp-examples. This library allows you to generate all (or one random) strings that match any regular expression! (With just a few limitations on what’s possible.) To install it yourself and have a quick play is dead easy:

> gem install regexp-examples
  Fetching: regexp-examples-1.3.1.gem (100%)
  Successfully installed regexp-examples-1.3.1
  Parsing documentation for regexp-examples-1.3.1
  Installing ri documentation for regexp-examples-1.3.1
  Done installing documentation for regexp-examples after 0 seconds
  1 gem installed
> pry
[1] pry(main)> require 'regexp-examples'
  => true
[2] pry(main)> ls /rexex/
  CoreExtensions::Regexp::Examples#methods: examples  random_example
  Regexp#methods: ==  ===  =~  casefold?  encoding  eql?  fixed_encoding?
    hash  inspect  match  named_captures  names  options  source  to_s  ~
[3] pry(main)> /quick|little|demo/.examples
  => ["quick", "little", "demo"]
[4] pry(main)> /\w{10}@(hotmail|gmail)\.com <--Backref! \1/.random_example
  => "cpRe_f2vMI@gmail.com <--Backref! gmail"

In this post, I will try to explain some of the core logic and techniques that I used to build this library. What is a regular expression, really? How/why is it always possible de-construct a regex, to list all possible strings that match it? How on Earth did I get back-references to work?!

All shall be revealed…

What Is A “Regular” Expression?

Regular Expression

noun, computing

a sequence of symbols and characters expressing a string or pattern to be searched for within a longer piece of text.

Perhaps the most confusing aspect of regular expressions comes from their formal definition, and the fact that several features in the regex language are not really “regular” at all! These “irregular” pieces of syntax are, in short (and by no coincidence!), the “illegal syntax” in my regexp-examples gem.

However, rather than mysteriously telling you what a regular expression isn’t, let’s try to explain what it is:

There are only really four (yes, that’s right, four!) pieces of syntax allowed in a “true” regular expression:

The “empty string”, usually denoted by: ε
Literal characters, e.g. /abc123/
The * repeater, e.g. /a*b*c*/
The | (“Or”) operator, e.g. /a|b|c/

Oh, and there’s also brackets - so maybe five pieces of syntax, if you want to count those as well!

Every other piece of syntax is really just a nice way to simplify writing out horrendously long, complicated combinations of the above. Let’s try a few examples:

/a+/     == /a|a*/
/a?/     == /ε|a/
/a{2,4}/ == /aa|aaa|aaaa/
/a{3,}/  == /aaaa*/
/[a-d]*/ == /(a|b|c|d)*/
/\d\n?/  == /(0|1|2|3|4|5|6|7|8|9)(ε|\n)

# Note: My use of the == operator here is not to be taken
# too literally... In ruby, Regexp equality is not based
# purely on what strings they match!
# http://ruby-doc.org/core/Regexp.html#method-i-3D-3D

…Hopefully, you get the idea. To put this another way, any regex that can’t be expressed in this way is not truly a “regular” expression!

An easy way to see whether or not this is the case is: Does (part of) the regex need to know its surrounding context, in order to determine a match? For example:

# How does "\b" know whether it lies on a word boundary?
/\bword\b/

# How does "^" know whether it lies at the start of a line?
/line1\n^line2/

# Or in general, how can any "look-ahead"/"look-behind"
# be regular?
/irregular (?=expression)/

These all need to know “what came before”, or “what comes next”, and are therefore not True Regular Expressions™. Hopefully this makes the common claim that “back-references are not regular” a little more obvious to understand: You need to know what the capture group matches before you can know what the back-reference matches (i.e. knowledge of context). So of course you cannot express such patterns using only those four symbols!

One final point to make, before we move on: There is only really one type of repeater in regex; the others are all nothing more than shorthand:

/a/  == /a{1}/
/a?/ == /a{0,1}/
/a*/ == /a{0,}/
/a+/ == /a{1,}/

Understanding this structure is at the very heart of my ruby gem; the whole library architecture depends on (and, for some occasional edge cases, is restricted by!) it.

All True Regular Expressions™ are built using this structure:

/Group-Repeater-Group-Repeater-Group-Repeater-.../

Where every group can, itself, be built using that same structure.

What?! Show me some examples!

I’m glad you asked. Consider the following:

/(this|that)+/
== /(this|that){1,}/
== /(t{1}h{1}i{1}s{1}|t{1}h{1}a{1}t{1}){1,}/

(Yuck! Thankfully we don’t normally need to write them out like this!…) But this lays the foundations for the main purpose of this blog post:

How To Parse A Regular Expression

Without getting bogged down in the nitty-gritty implementation details of parsing, let’s dive straight in and look at the internal objects generated by RegexpExamples::Parser:

# After placing `binding.pry` at a crucial point, for inspection:
> /a*|b+/.random_example

[1] pry(RegexpExamples)> repeaters
=> [#<RegexpExamples::OrGroup:0x00000001598020
  @repeaters_list=
   [[#<RegexpExamples::StarRepeater:0x000000015983e0
      @group=#<RegexpExamples::SingleCharGroup:0x000000015984f8 @char="a", @ignorecase=false>,
      @max_repeats=2,
      @min_repeats=0>],
    [#<RegexpExamples::PlusRepeater:0x00000001598110
      @group=#<RegexpExamples::SingleCharGroup:0x00000001598188 @char="b", @ignorecase=false>,
      @max_repeats=3,
      @min_repeats=1>]]>]

This may look complicated, but it’s essentially not much different to what I described above. There is only one key additional thing to consider.

In order to avoid problems with infinity, we must restrict repeaters like * and + to have an upper limit. Taken straight from the gem’s README:

max_repeater_variance (default = 2) restricts how many examples to return for each repeater. For example:

.* is equivalent to .{0,2}

.+ is equivalent to .{1,3}

.{2,} is equivalent to .{2,4}

.{,3} is equivalent to .{0,2}

.{3,8} is equivalent to .{3,5}

Or, in other words, the above regex of /a*|b+/ has been interpreted as equivalent to the following:

/(a{0,2}|b{1,3}){1}/

Like I said above: /Group-Repeater-Group-Repeater-Group-Repeater-.../

How To Generate Examples From A Regular Expression

So, we have our parsed regex. All that remains is to transform this into its possible strings. The trick to this is that all groups and repeaters are given a special method: #result. These results are then built up, piece by piece, to form the full strings that match the regex. Let’s take the above example, one step at a time:

The SingleCharGroup ("a") has one possible result: ["a"]
Therefore the StarRepeater has three possible results: ["", "a", "aa"]
Similarly, SingleCharGroup ("b") has one possible result: ["b"]
Therefore, PlusRepeater has three possible results: ["b", "bb", "bbb"]
Next, the OrGroup simply concatenates these arrays of possible results, i.e. it has six possible results: ["", "a", "aa", "b", "bb", "bbb"]
And finally, the top level OneTimeRepeater just returns these same values.

And there you have it, for a fairly simple example! Let’s look at one more, to demonstrate perhaps the most important method in the whole gem:

# Again, placing `binding.pry` at the same crucial point:
> /a+|b+|c+/.examples

[1] pry(RegexpExamples)> repeaters
=> [#<RegexpExamples::OrGroup:0x000000013fc630
  @repeaters_list=
   [[#<RegexpExamples::PlusRepeater:0x000000013fcc70
      @group=#<RegexpExamples::SingleCharGroup:0x000000013fcd10 @char="a", @ignorecase=false>,
      @max_repeats=3,
      @min_repeats=1>],
    [#<RegexpExamples::PlusRepeater:0x000000013fca18
      @group=#<RegexpExamples::SingleCharGroup:0x000000013fcae0 @char="b", @ignorecase=false>,
      @max_repeats=3,
      @min_repeats=1>],
    [#<RegexpExamples::PlusRepeater:0x000000013fc770
      @group=#<RegexpExamples::SingleCharGroup:0x000000013fc7e8 @char="c", @ignorecase=false>,
      @max_repeats=3,
      @min_repeats=1>]]>]

Once again, we make use of PlusRepeater#result and SingleCharGroup#result to build up the final answer from each “partial result”.

However, in this case we end up with the following:

[["a", "aa", "aaa"], ["b", "bb", "bbb"], ["c", "cc", "ccc"]]

Where each of those inner arrays is the result of each PlusRepeater. We need to make one more step: Find all possible results, from joining one element from each array, to form a “final result” string. Enter the magic glue that holds this whole thing together:

# Given an array of arrays of strings, returns all possible perutations
# for strings, created by joining one element from each array
#
# For example:
# permutations_of_strings [ ['a'], ['b'], ['c', 'd', 'e'] ] #=> ['abc', 'abd', 'abe']
# permutations_of_strings [ ['a', 'b'], ['c', 'd'] ] #=> [ 'ac', 'ad', 'bc', 'bd' ]
def self.permutations_of_strings(arrays_of_strings)
  first = arrays_of_strings.shift
  return first if arrays_of_strings.empty?
  first.product( permutations_of_strings(arrays_of_strings) )
end

_{*I’ve actually simplified this method slightly, to avoid confusion. The
real deal can be found
here.}

And so, after applying this method to the above array, we end up with:

["abc", "abcc", "abccc", "abbc", "abbcc", .....]

This method gets used a lot, when dealing with more complicated regexes! It is the magic function that allows patterns to be made arbitrarily complicated, with unlimited nesting of groups and so on.

So, there you have it! Now you understand all about how to generate examples from regular expressions, right?…

Oh, but…

How do you deal with escaped characters, like \d, \W, etc?
What about regexp options (ignorecase, multiline, extended form)?
What about unicode characters, control codes, named properties, and so on?
How on earth do you correctly parse all of the possible syntax in character sets, such as:

/[abc]/.examples
/[a-z]/.examples
/[^\d\ba-c]/.examples
/[[:alpha:]&&[a-c]]/.examples

…And I’m barely getting started here! There is a huge range of syntax to consider!

To cut a long story short: parsing is complicated!! However, all the basic principles discussed above still apply.

There is just one final piece of the puzzle that I have mostly avoided up until this point: back-references.

As discussed earlier, back-references are not regular, as they require knowledge of context. They are not strictly possible to fully support with this gem! (And indeed, there are some rare edge cases where my solution does not work.)

But, as promised, all shall be revealed…

How to generate examples with back-references

The important thing to recognise here is that you cannot know what the back-references need to match, until after the rest of the regex example has been generated. For example, consider the following:

/(a|b)\1/.random_example

You cannot possibly know whether the \1 is meant to be an "a" or a "b", until after the capture group’s “partial example” is chosen!

The solution? We cheat, and use a place-holder - then substitute the correct pattern back in later!

The pattern I chose is: __X__, where X is the name of your back-reference (in this case, "1").

There is a lot of intricate logic involved in actually keeping track of the results of these capture groups (perhaps the topic for a follow-up blog post?), so let’s gloss over this detail for now. So in summary, examples for the above regex are calculated as follows:

The SingleCharGroup ("a") has one possible result: ["a"]
The SingleCharGroup ("b") has one possible result: ["b"]
The OrGroup has two possible results: ["a", "b"]
The MultiGroup with group_id=1 has two possible results: ["a", "b"]
The BackReferenceGroup has one possible result: ["__1__"]
This gives us a final array of possible results: [["a", "b"], ["__1__"]]
After applying the permutations_of_strings method, this gives us two “final results”: ["a__1__", "b__1__"]
We now do one final step: Apply a #substitute_backreferences method on each string, to reveal the true strings that match the original regex: ["aa", "bb"]

And now finally, young Padawan, you are ready to see the actual implementation of Regexp#examples:

def examples
  # map_results basically just calls permutations_of_strings(repeater.result),
  # on each repeater
  full_examples = RegexpExamples.map_results(
    RegexpExamples::Parser.new(source, options).parse
  )
  RegexpExamples::BackReferenceReplacer.new.substitute_backreferences(full_examples)

_{*Once again, I’ve been naughty and shown you a slightly simplified version,
to avoid confusion. See the real thing over
here.}

One Final Example

I’ll leave you with one final example, showing the true power of this gem.

Question: What the hell does this ridiculous regex match?! (Side note: It’s usually a bad idea to validate email addresses like this. If you want to ensure the address is correct, just send a confirmation email!)

Answer: (And on my average machine, it took ~0.01 seconds to generate this example string!!)

# Yes, this line of code is ridiculous :D
pry(main)> email_regex = %r{(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])}
pry(main)> email_regex.random_example
  => "cec.y.1@gh.jd"