previous-exams/2021-midterm/m1.md

-Use the following commands to make a fresh clone of your repository:
-
-```
-git clone -b m1 git@gitlab.epfl.ch:lamp/student-repositories-s21/cs206-GASPAR.git m1
-```
-
-## Useful links
-
-  * [A guide to the Scala parallel collections](https://docs.scala-lang.org/overviews/parallel-collections/overview.html)
-  * [The API documentation of the Scala parallel collections](https://www.javadoc.io/doc/org.scala-lang.modules/scala-parallel-collections_2.13/latest/scala/collection/index.html)
-  * [The API documentation of the Scala standard library](https://www.scala-lang.org/files/archive/api/2.13.4)
-  * [The API documentation of the Java standard library](https://docs.oracle.com/en/java/javase/15/docs/api/index.html)
-
-**If you have issues with the IDE, try [reimporting the
-build](https://gitlab.epfl.ch/lamp/cs206/-/blob/master/labs/example-lab.md#ide-features-like-type-on-hover-or-go-to-definition-do-not-work),
-if you still have problems, use `compile` in sbt instead.**
-
-## Exercise
-
-Given the following sequential implementation of a function that computes the sequence of rolling averages, your task will be to complete and optimize a parallel version of this code.
-
-```scala
-/** Compute the rolling average of array.
- *
- *  For an array `arr = Arr(x1, x2, x3, ..., xn)` the result is
- *  `Arr(x1 / 1, (x1 + x2) / 2, (x1 + x2 + x3) / 3, ..., (x1 + x2 + x3 + ... + xn) / n)`
- */
-def rollingAveragesSequential(arr: Array[Int]): Array[Double] =
-  // Transform all numbers to fractions with denominator 1
-  val arr1 = arr.map(x => Frac(x, 1))
-  // Compute the rolling average keeping the sum of all elements in the numerator and the count of elements in the denominator.
-  val arr2 = arr1.scan(Frac(0, 0))((acc, x) => Frac(acc.numerator + x.numerator, acc.denominator + x.denominator))
-  // Transform fractions to Doubles
-  arr2.map(frac => frac.toDouble)
-  // Drop the extra initial element that was added by the scan
-  arr3.tail
-```
-
- This implementation has some issues:
- - It does not use parallelism
- - Creates two intermediate arrays by calling `map`
- - Creates an extra intermediate arrays by calling `tail`
- - Scan returns an extra element we do not need
-
- We want to parallelize and avoid the creation of the extra arrays.
- As we are calling a `scan` the natural operations we need are `upsweep` and `downsweep`.
- It is possible specialize those operations for our problem by letting those operations do the mapping.
- It is also possible to change those operations to not generate the first element.
-
-We give you a version of `rollingAveragesSequential` that partially implements the parallelization using `upsweep` and `downsweep`.
-
- Your tasks in the exercise will be to:
- - TASK 1: Implement the parallelization of `upsweep` and `downsweep`
- - TASK 2: Remove the calls to the `map`
- - TASK 3: Remove the call to `tail`
-
- You can get partial points for solving part of the tasks.
- The order of the tasks is a suggestion, you may do them in any order if that is simpler for you.
-
-Look at the `Lib` trait to find the definitions of functions and classes you can use (or already used).
-In this question we use a `Arr` array class instead of the normal `Array`. You may assume that this class has the same performance characteristics as the normal array. `Arr` provides only a limited set of operations.

previous-exams/2021-midterm/m1/.gitignore

-# General
-*.DS_Store
-*.swp
-*~
-
-# Dotty
-*.class
-*.tasty
-*.hasTasty
-
-# sbt
-target/
-
-# IDE
-.bsp
-.bloop
-.metals
-.vscode
-
-# datasets
-stackoverflow-grading.csv
-wikipedia-grading.dat

previous-exams/2021-midterm/m1/assignment.sbt

-// Student tasks (i.e. submit, packageSubmission)
-enablePlugins(StudentTasks)

previous-exams/2021-midterm/m1/build.sbt

-course := "midterm"
-assignment := "m1"
-scalaVersion := "3.0.0-RC1"
-scalacOptions ++= Seq("-language:implicitConversions", "-deprecation")
-
-libraryDependencies += "org.scalameta" %% "munit" % "0.7.22"
-
-val MUnitFramework = new TestFramework("munit.Framework")
-testFrameworks += MUnitFramework
-// Decode Scala names
-testOptions += Tests.Argument(MUnitFramework, "-s")
-testSuite := "m1.M1Suite"

previous-exams/2021-midterm/m1/project/FilteringReporterPlugin.scala

-package sbt // To access the private[sbt] compilerReporter key
-package filteringReporterPlugin
-
-import Keys._
-import ch.epfl.lamp._
-
-object FilteringReporterPlugin extends AutoPlugin {
-  override lazy val projectSettings = Seq(
-    // Turn off warning coming from scalameter that we cannot fix without changing scalameter
-    compilerReporter in (Compile, compile) ~= { reporter => new FilteringReporter(reporter) }
-  )
-}
-
-class FilteringReporter(reporter: xsbti.Reporter) extends xsbti.Reporter {
-
-  def reset(): Unit = reporter.reset()
-  def hasErrors: Boolean = reporter.hasErrors
-  def hasWarnings: Boolean = reporter.hasWarnings
-  def printSummary(): Unit = reporter.printSummary()
-  def problems: Array[xsbti.Problem] = reporter.problems
-
-  def log(problem: xsbti.Problem): Unit = {
-    if (!problem.message.contains("An existential type that came from a Scala-2 classfile cannot be"))
-      reporter.log(problem)
-  }
-
-  def comment(pos: xsbti.Position, msg: String): Unit =
-    reporter.comment(pos, msg)
-
-  override def toString = s"CollectingReporter($reporter)"
-}

previous-exams/2021-midterm/m1/project/MOOCSettings.scala

-package ch.epfl.lamp
-
-import sbt._
-import sbt.Keys._
-
-/**
- * Coursera uses two versions of each assignment. They both have the same assignment key and part id but have
- * different item ids.
- *
- * @param key Assignment key
- * @param partId Assignment partId
- * @param itemId Item id of the non premium version
- * @param premiumItemId Item id of the premium version (`None` if the assignment is optional)
- */
-case class CourseraId(key: String, partId: String, itemId: String, premiumItemId: Option[String])
-
-/**
-  * Settings shared by all assignments, reused in various tasks.
-  */
-object MOOCSettings extends AutoPlugin {
-
-  override def requires = super.requires && filteringReporterPlugin.FilteringReporterPlugin
-
-  object autoImport {
-    val course = SettingKey[String]("course")
-    val assignment = SettingKey[String]("assignment")
-    val options = SettingKey[Map[String, Map[String, String]]]("options")
-    val courseraId = settingKey[CourseraId]("Coursera-specific information identifying the assignment")
-    val testSuite = settingKey[String]("Fully qualified name of the test suite of this assignment")
-      .withRank(KeyRanks.Invisible)
-    // Convenient alias
-    type CourseraId = ch.epfl.lamp.CourseraId
-    val CourseraId = ch.epfl.lamp.CourseraId
-  }
-
-  import autoImport._
-
-  override val globalSettings: Seq[Def.Setting[_]] = Seq(
-    // supershell is verbose, buggy and useless.
-    useSuperShell := false
-  )
-
-  override val projectSettings: Seq[Def.Setting[_]] = Seq(
-    parallelExecution in Test := false,
-    // Report test result after each test instead of waiting for every test to finish
-    logBuffered in Test := false,
-    name := s"${course.value}-${assignment.value}"
-  )
-}

previous-exams/2021-midterm/m1/project/StudentTasks.scala

-package ch.epfl.lamp
-
-import sbt._
-import Keys._
-
-// import scalaj.http._
-import java.io.{File, FileInputStream, IOException}
-import org.apache.commons.codec.binary.Base64
-// import play.api.libs.json.{Json, JsObject, JsPath}
-import scala.util.{Failure, Success, Try}
-
-/**
-  * Provides tasks for submitting the assignment
-  */
-object StudentTasks extends AutoPlugin {
-
-  override def requires = super.requires && MOOCSettings
-
-  object autoImport {
-    val packageSourcesOnly = TaskKey[File]("packageSourcesOnly", "Package the sources of the project")
-    val packageBinWithoutResources = TaskKey[File]("packageBinWithoutResources", "Like packageBin, but without the resources")
-    val packageSubmissionZip = TaskKey[File]("packageSubmissionZip")
-    val packageSubmission = inputKey[Unit]("package solution as an archive file")
-
-    lazy val Grading = config("grading") extend(Runtime)
-  }
-
-
-  import autoImport._
-  import MOOCSettings.autoImport._
-
-  override lazy val projectSettings = Seq(
-    packageSubmissionSetting,
-    fork := true,
-    connectInput in run := true,
-    outputStrategy := Some(StdoutOutput),
-  ) ++
-    packageSubmissionZipSettings ++
-    inConfig(Grading)(Defaults.testSettings ++ Seq(
-      unmanagedJars += file("grading-tests.jar"),
-
-      definedTests := (definedTests in Test).value,
-      internalDependencyClasspath := (internalDependencyClasspath in Test).value
-    ))
-
-
-  /** **********************************************************
-    * SUBMITTING A SOLUTION TO COURSERA
-    */
-
-  val packageSubmissionZipSettings = Seq(
-    packageSubmissionZip := {
-      val submission = crossTarget.value / "submission.zip"
-      val sources = (packageSourcesOnly in Compile).value
-      val binaries = (packageBinWithoutResources in Compile).value
-      IO.zip(Seq(sources -> "sources.zip", binaries -> "binaries.jar"), submission, None)
-      submission
-    },
-    artifactClassifier in packageSourcesOnly := Some("sources"),
-    artifact in (Compile, packageBinWithoutResources) ~= (art => art.withName(art.name + "-without-resources"))
-  ) ++
-  inConfig(Compile)(
-    Defaults.packageTaskSettings(packageSourcesOnly, Defaults.sourceMappings) ++
-    Defaults.packageTaskSettings(packageBinWithoutResources, Def.task {
-      val relativePaths =
-        (unmanagedResources in Compile).value.flatMap(Path.relativeTo((unmanagedResourceDirectories in Compile).value)(_))
-      (mappings in (Compile, packageBin)).value.filterNot { case (_, path) => relativePaths.contains(path) }
-    })
-  )
-
-  val maxSubmitFileSize = {
-    val mb = 1024 * 1024
-    10 * mb
-  }
-
-  /** Check that the jar exists, isn't empty, isn't crazy big, and can be read
-    * If so, encode jar as base64 so we can send it to Coursera
-    */
-  def prepareJar(jar: File, s: TaskStreams): String = {
-    val errPrefix = "Error submitting assignment jar: "
-    val fileLength = jar.length()
-    if (!jar.exists()) {
-      s.log.error(errPrefix + "jar archive does not exist\n" + jar.getAbsolutePath)
-      failSubmit()
-    } else if (fileLength == 0L) {
-      s.log.error(errPrefix + "jar archive is empty\n" + jar.getAbsolutePath)
-      failSubmit()
-    } else if (fileLength > maxSubmitFileSize) {
-      s.log.error(errPrefix + "jar archive is too big. Allowed size: " +
-        maxSubmitFileSize + " bytes, found " + fileLength + " bytes.\n" +
-        jar.getAbsolutePath)
-      failSubmit()
-    } else {
-      val bytes = new Array[Byte](fileLength.toInt)
-      val sizeRead = try {
-        val is = new FileInputStream(jar)
-        val read = is.read(bytes)
-        is.close()
-        read
-      } catch {
-        case ex: IOException =>
-          s.log.error(errPrefix + "failed to read sources jar archive\n" + ex.toString)
-          failSubmit()
-      }
-      if (sizeRead != bytes.length) {
-        s.log.error(errPrefix + "failed to read the sources jar archive, size read: " + sizeRead)
-        failSubmit()
-      } else encodeBase64(bytes)
-    }
-  }
-
-  /** Task to package solution to a given file path */
-  lazy val packageSubmissionSetting = packageSubmission := {
-    val args: Seq[String] = Def.spaceDelimited("[path]").parsed
-    val s: TaskStreams = streams.value // for logging
-    val jar = (packageSubmissionZip in Compile).value
-
-    val base64Jar = prepareJar(jar, s)
-
-    val path = args.headOption.getOrElse((baseDirectory.value / "submission.jar").absolutePath)
-    scala.tools.nsc.io.File(path).writeAll(base64Jar)
-  }
-
-/*
-  /** Task to submit a solution to coursera */
-  val submit = inputKey[Unit]("submit solution to Coursera")
-  lazy val submitSetting = submit := {
-    // Fail if scalafix linting does not pass.
-    scalafixLinting.value
-
-    val args: Seq[String] = Def.spaceDelimited("<arg>").parsed
-    val s: TaskStreams = streams.value // for logging
-    val jar = (packageSubmissionZip in Compile).value
-
-    val assignmentDetails =
-      courseraId.?.value.getOrElse(throw new MessageOnlyException("This assignment can not be submitted to Coursera because the `courseraId` setting is undefined"))
-    val assignmentKey = assignmentDetails.key
-    val courseName =
-      course.value match {
-        case "capstone" => "scala-capstone"
-        case "bigdata"  => "scala-spark-big-data"
-        case other      => other
-      }
-
-    val partId = assignmentDetails.partId
-    val itemId = assignmentDetails.itemId
-    val premiumItemId = assignmentDetails.premiumItemId
-
-    val (email, secret) = args match {
-      case email :: secret :: Nil =>
-        (email, secret)
-      case _ =>
-        val inputErr =
-          s"""|Invalid input to `submit`. The required syntax for `submit` is:
-              |submit <email-address> <submit-token>
-              |
-              |The submit token is NOT YOUR LOGIN PASSWORD.
-              |It can be obtained from the assignment page:
-              |https://www.coursera.org/learn/$courseName/programming/$itemId
-              |${
-                premiumItemId.fold("") { id =>
-                  s"""or (for premium learners):
-                     |https://www.coursera.org/learn/$courseName/programming/$id
-                   """.stripMargin
-                }
-              }
-          """.stripMargin
-        s.log.error(inputErr)
-        failSubmit()
-    }
-
-    val base64Jar = prepareJar(jar, s)
-    val json =
-      s"""|{
-          |   "assignmentKey":"$assignmentKey",
-          |   "submitterEmail":"$email",
-          |   "secret":"$secret",
-          |   "parts":{
-          |      "$partId":{
-          |         "output":"$base64Jar"
-          |      }
-          |   }
-          |}""".stripMargin
-
-    def postSubmission[T](data: String): Try[HttpResponse[String]] = {
-      val http = Http("https://www.coursera.org/api/onDemandProgrammingScriptSubmissions.v1")
-      val hs = List(
-        ("Cache-Control", "no-cache"),
-        ("Content-Type", "application/json")
-      )
-      s.log.info("Connecting to Coursera...")
-      val response = Try(http.postData(data)
-                         .headers(hs)
-                         .option(HttpOptions.connTimeout(10000)) // scalaj default timeout is only 100ms, changing that to 10s
-                         .asString) // kick off HTTP POST
-      response
-    }
-
-    val connectMsg =
-      s"""|Attempting to submit "${assignment.value}" assignment in "$courseName" course
-          |Using:
-          |- email: $email
-          |- submit token: $secret""".stripMargin
-    s.log.info(connectMsg)
-
-    def reportCourseraResponse(response: HttpResponse[String]): Unit = {
-      val code = response.code
-      val respBody = response.body
-
-       /* Sample JSON response from Coursera
-      {
-        "message": "Invalid email or token.",
-        "details": {
-          "learnerMessage": "Invalid email or token."
-        }
-      }
-      */
-
-      // Success, Coursera responds with 2xx HTTP status code
-      if (response.is2xx) {
-        val successfulSubmitMsg =
-          s"""|Successfully connected to Coursera. (Status $code)
-              |
-                |Assignment submitted successfully!
-              |
-                |You can see how you scored by going to:
-              |https://www.coursera.org/learn/$courseName/programming/$itemId/
-              |${
-            premiumItemId.fold("") { id =>
-              s"""or (for premium learners):
-                 |https://www.coursera.org/learn/$courseName/programming/$id
-                       """.stripMargin
-            }
-          }
-              |and clicking on "My Submission".""".stripMargin
-        s.log.info(successfulSubmitMsg)
-      }
-
-      // Failure, Coursera responds with 4xx HTTP status code (client-side failure)
-      else if (response.is4xx) {
-        val result = Try(Json.parse(respBody)).toOption
-        val learnerMsg = result match {
-          case Some(resp: JsObject) =>
-            (JsPath \ "details" \ "learnerMessage").read[String].reads(resp).get
-          case Some(x) => // shouldn't happen
-            "Could not parse Coursera's response:\n" + x
-          case None =>
-            "Could not parse Coursera's response:\n" + respBody
-        }
-        val failedSubmitMsg =
-          s"""|Submission failed.
-              |There was something wrong while attempting to submit.
-              |Coursera says:
-              |$learnerMsg (Status $code)""".stripMargin
-        s.log.error(failedSubmitMsg)
-      }
-
-      // Failure, Coursera responds with 5xx HTTP status code (server-side failure)
-      else if (response.is5xx) {
-        val failedSubmitMsg =
-          s"""|Submission failed.
-              |Coursera seems to be unavailable at the moment (Status $code)
-              |Check https://status.coursera.org/ and try again in a few minutes.
-           """.stripMargin
-        s.log.error(failedSubmitMsg)
-      }
-
-      // Failure, Coursera repsonds with an unexpected status code
-      else {
-        val failedSubmitMsg =
-          s"""|Submission failed.
-              |Coursera replied with an unexpected code (Status $code)
-           """.stripMargin
-        s.log.error(failedSubmitMsg)
-      }
-    }
-
-    // kick it all off, actually make request
-    postSubmission(json) match {
-      case Success(resp) => reportCourseraResponse(resp)
-      case Failure(e) =>
-        val failedConnectMsg =
-          s"""|Connection to Coursera failed.
-              |There was something wrong while attempting to connect to Coursera.
-              |Check your internet connection.
-              |${e.toString}""".stripMargin
-        s.log.error(failedConnectMsg)
-    }
-
-   }
-*/
-
-  def failSubmit(): Nothing = {
-    sys.error("Submission failed")
-  }
-
-  /**
-    * *****************
-    * DEALING WITH JARS
-    */
-  def encodeBase64(bytes: Array[Byte]): String =
-    new String(Base64.encodeBase64(bytes))
-}

previous-exams/2021-midterm/m1/project/build.properties

-sbt.version=1.4.7

previous-exams/2021-midterm/m1/project/buildSettings.sbt

-// Used for Coursera submission (StudentPlugin)
-// libraryDependencies += "org.scalaj" %% "scalaj-http" % "2.4.2"
-// libraryDependencies += "com.typesafe.play" %% "play-json" % "2.7.4"
-// Used for Base64 (StudentPlugin)
-libraryDependencies += "commons-codec" % "commons-codec" % "1.10"

previous-exams/2021-midterm/m1/project/plugins.sbt

-// addSbtPlugin("org.scala-js" % "sbt-scalajs" % "0.6.28")
-addSbtPlugin("ch.epfl.lamp" % "sbt-dotty" % "0.5.3")

previous-exams/2021-midterm/m1/src/main/scala/m1/Lib.scala

-package m1
-
-////////////////////////////////////////
-// NO NEED TO MODIFY THIS SOURCE FILE //
-////////////////////////////////////////
-
-trait Lib {
-
-  /** If an array has `n` elements and `n < THRESHOLD`, then it should be processed sequentially */
-  final val THRESHOLD: Int = 33
-
-  /** Compute the two values in parallel
-   *
-   *  Note: Most tests just compute those two sequentially to make any bug simpler to debug
-   */
-  def parallel[T1, T2](op1: => T1, op2: => T2): (T1, T2)
-
-  /** A limited array. It only contains the required operations for this exercise. */
-  trait Arr[T] {
-    /** Get the i-th element of the array (0-based) */
-    def apply(i: Int): T
-    /** Update the i-th element of the array with the given value (0-based) */
-    def update(i: Int, x: T): Unit
-    /** Number of elements in this array */
-    def length: Int
-    /** Create a copy of this array without the first element */
-    def tail: Arr[T]
-    /** Create a copy of this array by mapping all the elements with the given function */
-    def map[U](f: T => U): Arr[U]
-  }
-
-  object Arr {
-    /** Create an array with the given elements */
-    def apply[T](xs: T*): Arr[T] = {
-      val arr: Arr[T] = Arr.ofLength(xs.length)
-      for i <- 0 until xs.length do arr(i) = xs(i)
-      arr
-    }
-
-    /** Create an array with the given length. All elements are initialized to `null`. */
-    def ofLength[T](n: Int): Arr[T] =
-      newArrOfLength(n)
-
-  }
-
-  /** Create an array with the given length. All elements are initialized to `null`. */
-  def newArrOfLength[T](n: Int): Arr[T]
-
-  /** A fractional number representing `numerator/denominator` */
-  case class Frac(numerator: Int, denominator: Int) {
-    def toDouble: Double = numerator.toDouble / denominator
-  }
-
-  /** Tree result of an upsweep operation. Specialized for `Frac` results. */
-  trait TreeRes { val res: Frac }
-  /** Leaf result of an upsweep operation. Specialized for `Frac` results. */
-  case class Leaf(from: Int, to: Int, res: Frac) extends TreeRes
-  /** Tree node result of an upsweep operation. Specialized for `Frac` results. */
-  case class Node(left: TreeRes, res: Frac, right: TreeRes) extends TreeRes
-}

previous-exams/2021-midterm/m1/src/main/scala/m1/M1.scala

-package m1
-
-
-trait M1 extends Lib {
-  // Functions and classes of Lib can be used in here
-
-  /** Compute the rolling average of array.
-   *
-   *  For an array `arr = Arr(x1, x2, x3, ..., xn)` the result is
-   *  `Arr(x1 / 1, (x1 + x2) / 2, (x1 + x2 + x3) / 3, ..., (x1 + x2 + x3 + ... + xn) / n)`
-   */
-  def rollingAveragesParallel(arr: Arr[Int]): Arr[Double] = {
-    if (arr.length == 0) return Arr.ofLength(0)
-
-    val out: Arr[Double] = Arr.ofLength(arr.length)
-    val tree = upsweep(arr, 0, arr.length)
-    downsweep(arr, Frac(0, 0), tree, out)
-    out
-  }
-
-  // No need to modify this
-  def scanOp(acc: Frac, x: Frac) =
-    Frac(acc.numerator + x.numerator, acc.denominator + x.denominator)
-
-
-  def upsweep(input: Arr[Int], from: Int, to: Int): TreeRes = {
-    if (to - from < THRESHOLD)
-      Leaf(from, to, reduceSequential(input, from + 1, to, Frac(input(from), 1)))
-    else {
-      val mid = from + (to - from)/2
-      val (tL, tR) = parallel(
-        upsweep(input, from, mid),
-        upsweep(input, mid, to)
-      )
-      Node(tL, scanOp(tL.res, tR.res), tR)
-    }
-  }
-
-
-  def downsweep(input: Arr[Int], a0: Frac, tree: TreeRes, output: Arr[Double]): Unit = {
-    tree match {
-      case Node(left, _, right) =>
-        parallel(
-          downsweep(input, a0, left, output),
-          downsweep(input, scanOp(a0, left.res), right, output)
-        )
-      case Leaf(from, to, _) =>
-        downsweepSequential(input, from, to, a0, output)
-    }
-  }
-
-
-  def downsweepSequential(input: Arr[Int], from: Int, to: Int, a0: Frac, output: Arr[Double]): Unit = {
-    if (from < to) {
-      var i = from
-      var a = a0
-      while (i < to) {
-        a = scanOp(a, Frac(input(i), 1))
-        output(i) = a.toDouble
-        i = i + 1
-      }
-    }
-  }
-
-
-  def reduceSequential(input: Arr[Int], from: Int, to: Int, a0: Frac): Frac = {
-    var a = a0
-    var i = from
-    while (i < to) {
-      a = scanOp(a, Frac(input(i), 1))
-      i = i + 1
-    }
-    a
-  }
-
-}