spark2

 whenever rdd contains tuple of 2 elements-- then its called pair rdd.


last line val result = sortedTotal.collect


here result is the local variable on the local machine not rdd

rdd--disttibuted-- on the cluster


unix timestamp--no of sec after 1st jan 1970


reduceByKey((x,y) => x+y)


here x, y works on two rows


instead of using map where we say (x,1) and doing reduceByKey later


map+reduceByKey ---result --rdd-- its transformation

=

countByValue--action--local variable


if that is final operation--then u can use countByValue..--result will be local variable and paralism won't happen if u do any tranformation.


map+reduceByKey--if u want to do transformation after this.


reduceByKey-- always works on value--we don't have to worries about keys.


mapValues--works on values only(if key is not changing and we will work on values only).


Comments

Post a Comment

Popular posts from this blog

scala-4

Scala: ----------------------------- The s interpolator It allows the use of variable directly in processing a string, when you prepend ‘s’ to it. val name = "John" println(s"Hello $name") //output: Hello John String interpolation can also process arbitrary expressions. println(s"2 + 3 = ${2 + 3}") //output: 2 + 3 = 5 The f interpolator It allows creating a formatted String, similar to printf in C language. While using ‘f’ interpolator, all variable references should be followed by the printf style format specifiers such as %d, %i, %f, etc. val height = 1.7d val name = "John" println(f"$name%s is $height%2.2f meters tall") //John is 1.70 meters tall The raw interpolator It is similar to s interpolator except that it performs no escaping of literals within a string. Output with s option: println(s"Result = \n a \n b") Result =  a  b Outout with raw option: println(raw"Result = \n a \n b") Result = \n a \n b Example 1:...
Read more