I’m aware that at the time of writing Spark 1.4.1 has been released but I used the 1.4.0 release\u00a0to match our current dev-cluster’s version.<\/p>\n
Install R<\/h3>\n
Download and install the R distribution for Mac OS X from https:\/\/cran.r-project.org\/bin\/macosx\/<\/a>\u00a0and install using ‘the’ regular method.<\/p>\n Download and install RStudio for your OS from\u00a0https:\/\/www.rstudio.com\/products\/rstudio\/download\/<\/a><\/p>\n Obviously, to actually run SparkR jobs, you will be needing a Spark cluster. In my case, I used a\u00a0Intel NUC-based 3-node ‘take it with you’ cluster provisioned with Apache Hadoop 2.7.0 and Spark 1.4.0. I have one Spark master called nucluster3 and three slaves called nucluster[1-3] (the master is also a slave).<\/p>\n I will not go into detail in this post on installing a development Spark cluster apart from stating that it should not give you any headache getting it up and running as long as you only edit SPARK_HOME\/conf\/spark-env.sh (add JAVA_HOME variable) and SPARK_HOME\/conf\/slaves (list all slave hostnames) and start the cluster on the master node<\/em>\u00a0(nucluster3 in our\u00a0case) by using start-master.sh and start-slaves.sh. You should than be greeted on http:\/\/nucluster3:8080<\/a>\u00a0by:<\/p>\n All’s set and done from the software perspective, so let’s head over to the setup and the actual running of R statements.<\/p>\n We need a file to work on. For this post, I used a file with zipcodes called postcode.csv. You may upload it to your HDFS cluster with HUE, but I used the hardcore way by first copying it to one of the nodes and from there, used the hadoop fs<\/strong> command to put it on HDFS:<\/p>\n “id”,”postcode”,”postcode_id”,”pnum”,”pchar”,”minnumber”,”maxnumber”,”numbertype”,”street”,”city”,”city_id”,”municipality”,”municipality_id”,”province”,”province_code”,”lat”,”lon”,”rd_x”,”rd_y”,”location_detail”,”changed_date”<\/em> I did not bother to strip or use the header in this case. It’s just there. So, you’re now ready to start your analysis! Fire up RStudio, you will be greeted by the familiar R prompt, accompanied by some warnings you may safely ignore for now.<\/p>\n Now a few things have to be set up. R needs to know where it can find the SparkR library. In my case I use the following command:<\/p>\n .libPaths(c(.libPaths(), ‘\/Users\/rutger\/Development\/spark-1.4.0-bin-hadoop2.6\/R\/lib’))<\/strong><\/p>\n No response will be given. Your R analysis will actually be an application running on the Spark cluster submitted by the sparkr-shell. So R(Studio) should know where it can find this command. Because a CSV file is to be read, the spark-csv package is also loaded:<\/p>\n Sys.setenv(PATH = paste(Sys.getenv(c(‘PATH’)), ‘\/Users\/rutger\/Development\/spark-1.4.0-bin-hadoop2.6\/bin’, sep=’:’))<\/strong><\/p>\n Sys.setenv(‘SPARKR_SUBMIT_ARGS’='”–packages” “com.databricks:spark-csv_2.11:1.1.0” “sparkr-shell”‘)<\/strong><\/p>\n Again, these commands should not give you any response unless you make a mistake. You are now ready to load the SparkR library:<\/p>\n library(SparkR)<\/strong><\/p>\n This will give you some feedback, something like<\/p>\n Attaching package: ‘SparkR’<\/em><\/p>\n Followed by some remarks about masked packages. They are harmless in our case:<\/p>\n The first thing we’ll have to do is create two contexts; the Spark context and the SparkRSQL context:<\/p>\n sc <- sparkR.init(master = “spark:\/\/nucluster3:7077″, appName=”SparkR”)<\/strong><\/p>\n sqlContext <- sparkRSQL.init(sc)<\/strong><\/p>\n The first command produces a fair amount of output but should complete without warnings, the second one should finish silently. Remember the postcodes I put on HDFS? We are now ready to load them into a dataframe. nucluster1 is the cluster’s namenode, by the way.<\/p>\n postcodes <- read.df(sqlContext, “hdfs:\/\/nucluster1:9000\/user\/root\/postcode.csv”, source = “com.databricks.spark.csv”)<\/strong><\/p>\n This command should finish successfully again with a fair amount of output.\u00a0You are now ready to do your R magic, but in the examples below I will stick to very<\/em> simple ones.<\/p>\n head(postcodes)<\/strong><\/p>\n count(postcodes)<\/strong><\/p>\n A slightly more difficult example is to count by one of the fields. In this case only the first few lines are returned<\/p>\n head(summarize(groupBy(postcodes, postcodes$C3), count = n(postcodes$C3)))<\/strong><\/p>\n Please note that without the ‘head’ command, nothing will actually be executed, only the return type will be displayed:<\/p>\n DataFrame[C3:string, count:bigint]\u00a0<\/em><\/p>\n One of the powerful features of this setup is the ability to use SQL in your commands. For this to work you’ll first have to create a temporary table definition.<\/p>\n registerTempTable(postcodes, “postcodes”)<\/strong><\/p>\n amsterdam <- sql(sqlContext, “SELECT C1,C8 FROM postcodes WHERE C9 = ‘Amsterdam'”)<\/strong> It may sound boring, but the first command will stay silent and the second one will return the query result (All Amsterdam zipcodes):<\/p>\n Of course, you are able to write much more cpu- and disk intensive queries. This totally bogus one put’s the cluster to work for a while:<\/p>\n count(sql(sqlContext, “SELECT p1.C1,p1.C8 FROM postcodes p1, postcodes p2 where p1.C1<p2.C1”))<\/strong><\/p>\n Note that all activity is done on the cluster<\/em>, not your own machine! Your machine is the job driver and only<\/em> serves as such. But you can view a lot of information about your job: take a look at http:\/\/localhost:4040\/jobs\/<\/a>\u00a0and find your own running one.<\/p>\n In the screenshot above you will see the ‘sort-of-cartesian-product’ I created hogging the system with a huge input set. Feel free to look around jobs, stages (note the beautiful DAG Visualization\u00a0Spark provides) and storage information. You may also\u00a0check out the Spark Master and note your running application named SparkR at\u00a0http:\/\/nucluster3:8080\/<\/a><\/p>\nInstall RStudio<\/h3>\n
Working Spark cluster<\/h3>\n
![\"aaeaaqaaaaaaaandaaaajgrmytdlyzqzlwi0ntutnguxzs04ntrhltzjztcxytmxyjiwma\"](\"http:\/\/whizzkit.nl\/wp-content\/uploads\/2016\/10\/AAEAAQAAAAAAAANdAAAAJGRmYTdlYzQzLWI0NTUtNGUxZS04NTRhLTZjZTcxYTMxYjIwMA-300x133.png\")
<\/a><\/p>\nAnalyze!<\/h2>\n
Setup<\/h3>\n
\n
\n395614,”7940XX”,79408888,7940,”XX”,4,12,”mixed”,”Troelstraplein”,”Meppel”,1082,”Meppel”,119,”Drenthe”,”DR”,”52.7047653217626″,”6.1977201775604″,”209781.52077777777777777778″,”524458.25733333333333333333″,”postcode”,”2014-04-10 13:20:28″<\/em><\/p>\n![\"aaeaaqaaaaaaaamlaaaajgqym2m3ndbjlwu1mwetndbkni1ioti4lwu3nmixmdiwyzdizq\"](\"http:\/\/whizzkit.nl\/wp-content\/uploads\/2016\/10\/AAEAAQAAAAAAAAMLAAAAJGQyM2M3NDBjLWU1MWEtNDBkNi1iOTI4LWU3NmIxMDIwYzdiZQ-300x134.png\")
<\/a><\/p>\n![\"aaeaaqaaaaaaaauraaaajdi5mwrlzguyltvlotktngnhoc04zdixltqyy2jhyjk3mji3oa\"](\"http:\/\/whizzkit.nl\/wp-content\/uploads\/2016\/10\/AAEAAQAAAAAAAAUrAAAAJDI5MWRlZGUyLTVlOTktNGNhOC04ZDIxLTQyY2JhYjk3MjI3OA-300x187.png\")
<\/a><\/p>\nPlain R commands<\/h3>\n
![\"aaeaaqaaaaaaaakoaaaajdvly2y4zjgyltiwotutngfmny05ndexltk0yzfkowiwywrmzq\"](\"http:\/\/whizzkit.nl\/wp-content\/uploads\/2016\/10\/AAEAAQAAAAAAAAKoAAAAJDVlY2Y4ZjgyLTIwOTUtNGFmNy05NDExLTk0YzFkOWIwYWRmZQ-300x180.png\")
<\/a><\/p>\n![\"aaeaaqaaaaaaaaskaaaajdiwyta3ndeyltnkotatnguxzc1hzjaylwy4mji1mzm0mdy4za\"](\"http:\/\/whizzkit.nl\/wp-content\/uploads\/2016\/10\/AAEAAQAAAAAAAASkAAAAJDIwYTA3NDEyLTNkOTAtNGUxZC1hZjAyLWY4MjI1MzM0MDY4ZA-300x71.png\")
<\/a><\/p>\n![\"aaeaaqaaaaaaaaxdaaaajgm2ndcxzdq5lthkntktngvims1injq0lwm3zmfhothlzmu1ng\"](\"http:\/\/whizzkit.nl\/wp-content\/uploads\/2016\/10\/AAEAAQAAAAAAAAXDAAAAJGM2NDcxZDQ5LThkNTktNGViMS1iNjQ0LWM3ZmFhOThlZmU1Ng-300x128.png\")
<\/a><\/p>\nSQL commands<\/h3>\n
\nhead(amsterdam)<\/strong><\/p>\n<\/a><\/p>\n![\"aaeaaqaaaaaaaaqfaaaajdbkodhlnjdilwmynzatngvkmi1hywqwltnjmduxntlhmzu1oq\"](\"http:\/\/whizzkit.nl\/wp-content\/uploads\/2016\/10\/AAEAAQAAAAAAAAQFAAAAJDBkODhlNjdiLWMyNzAtNGVkMi1hYWQwLTNjMDUxNTlhMzU1OQ-300x117.png\")
<\/a><\/p>\n
![\"aaeaaqaaaaaaaaahaaaajgy2nzk3mzyxltmwntytnge4zi05mta2ltnkmwfjytzknwvioq\"](\"http:\/\/whizzkit.nl\/wp-content\/uploads\/2016\/10\/AAEAAQAAAAAAAAaHAAAAJGY2Nzk3MzYxLTMwNTYtNGE4Zi05MTA2LTNkMWFjYTZkNWViOQ-300x24.png\")
<\/a><\/p>\n