typestar

word_freq.scala in Scala

A whole report: normalize a passage, count its words, rank and print them.

// A word-frequency report over an inline passage.

val passage: String =
  """Sailors watch the sky, because the sky warns the sailors.
    |A red sky at night can delight a sailor; a red sky in the
    |morning will warn the same sailor. Watch the sky, watch the
    |wind, and the sea will rarely surprise you.""".stripMargin

// Lowercasing first means Sky and sky land in the same bucket.
// Splitting on runs of non-letters drops punctuation and newlines.
def words(text: String): List[String] =
  text.toLowerCase.split("[^a-z]+").filter(_.nonEmpty).toList

// Two words carry no signal, so they are dropped from the report.
val stopWords: Set[String] = Set("the", "a", "and", "is", "in", "at", "you")

// groupBy(identity) buckets equal words; the size of a bucket is
// that word's count.
def tally(items: List[String]): Map[String, Int] =
  items.groupBy(identity).view.mapValues(_.size).toMap

@main def wordFreq(): Unit =
  val counted = tally(words(passage).filterNot(stopWords.contains))
  // Sorting by the negated count puts the most frequent first, and
  // the word itself breaks ties alphabetically.
  val ranked = counted.toList.sortBy((word, count) => (-count, word))
  val distinct = counted.size
  val total = counted.values.sum

  println(f"${"word"}%-12s${"count"}%6s${"share"}%8s")
  println("-" * 26)
  for (word, count) <- ranked.take(8) do
    val share = 100.0 * count / total
    println(f"$word%-12s$count%6d$share%7.1f%%")
  println("-" * 26)
  println(f"${"total"}%-12s$total%6d")
  println(f"$distinct distinct words kept out of ${words(passage).size}")

How it works

  1. text.toLowerCase.split("[^a-z]+") folds away case and punctuation in one pass.
  2. items.groupBy(identity).view.mapValues(_.size).toMap turns the word list into counts.
  3. sortBy((word, count) => (-count, word)) ranks by count then alphabetically, and $word%-12s$count%6d aligns the columns.

Keywords and builtins used here

The run, in numbers

Lines
37
Characters to type
1538
Tokens
262
Three-star pace
75 tpm

At the three-star pace of 75 tokens a minute, this run takes about 210 seconds.

Type this snippet

Step 1 of 3 in Encore, step 25 of 27 in Language basics.

← Previous Next →